Public bug reported:
We are investigating the possibilities of migrating from OVS to OVN.
Current state of the migration flow itself we have can be seen here
https://github.com/Mirantis/rockoon/blob/aef29880203481fb6383da09acb5046747f51137/rockoon/cli/ovs_ovn_migration.py
What we found is that after the migration is complete, the pre-existing
routers (created on OVS before migration) become unmanageable. For
example, one can't delete them via neutron API, the response is HTTP500
with the following error/trace in logs:
31276:2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource [None
req-9e728c4e-349a-4c15-b15b-4cbd0fe84da1 cffbb8f11d5f47d49f5ef63114ca04b6
71b11698a0aa4adb9974f825983b8e1a - - default default] delete failed: No
details.: neutron_lib.callbacks.exceptions.CallbackFailure: Callback
neutron.services.ovn_l3.service_providers.driver_controller.DriverController._clear_router_provider-649091
failed with "'ha'"
31277-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource Traceback (most
recent call last):
31278-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource File
"/var/lib/openstack/lib/python3.10/site-packages/neutron/api/v2/resource.py",
line 98, in resource
31279-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource result =
method(request=request, **args)
31280-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource File
"/var/lib/openstack/lib/python3.10/site-packages/neutron/api/v2/base.py", line
570, in delete
31281-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource return
self._delete(request, id, **kwargs)
31282-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource File
"/var/lib/openstack/lib/python3.10/site-packages/neutron_lib/db/api.py", line
137, in wrapped
31283-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource with
excutils.save_and_reraise_exception():
31284-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource File
"/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line
227, in __exit__
31285-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource
self.force_reraise()
31286-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource File
"/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line
200, in force_reraise
--
31342-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource File
"/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line
227, in __exit__
31343-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource
self.force_reraise()
31344-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource File
"/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line
200, in force_reraise
31345-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource raise
self.value
31346-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource File
"/var/lib/openstack/lib/python3.10/site-packages/neutron_lib/db/utils.py", line
104, in _wrapped
31347-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource return
function(*args, **kwargs)
31348-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource File
"/var/lib/openstack/lib/python3.10/site-packages/neutron_lib/callbacks/manager.py",
line 168, in publish
31349-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource raise
exceptions.CallbackFailure(errors=errors)
31350-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource
neutron_lib.callbacks.exceptions.CallbackFailure: Callback
neutron.services.ovn_l3.service_providers.driver_controller.DriverController._clear_router_provider-649091
failed with "'ha'"
31351-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource
31352:2025-02-25 11:39:26.913 28 INFO neutron.wsgi [None
req-9e728c4e-349a-4c15-b15b-4cbd0fe84da1 cffbb8f11d5f47d49f5ef63114ca04b6
71b11698a0aa4adb9974f825983b8e1a - - default default] 192.168.238.166 "DELETE
/v2.0/routers/bba5be48-2721-4ab2-840a-1cff81148ef2 HTTP/1.1" status: 500 len:
368 time: 5.0718703
Also, after migration one of ovn maintenance tasks (this is Caracal
specific, this code is already removed from master) keeps failing with
025-02-28T10:54:44.153763722Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics [None req-5a85f113-4549-4dca-b768-3e3cc4769f28 - - - - - -]
Failed to call periodic
'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.add_provider_resource_association_to_routers'
(it runs every 5.00 seconds):
neutron_lib.objects.exceptions.NeutronDbObjectDuplicateEntry: Failed to create
a duplicate ProviderResourceAssociation: for attribute(s) ['resource_id'] with
value(s) 4d9eb917-f05a-4590-a569-42cb59c52bab
2025-02-28T10:54:44.153824676Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics Traceback (most recent call last):
2025-02-28T10:54:44.153830586Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics File
"/var/lib/openstack/lib/python3.10/site-packages/sqlalchemy/engine/base.py",
line 1910, in _execute_context
2025-02-28T10:54:44.153834724Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics self.dialect.do_execute(
2025-02-28T10:54:44.153838399Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics File
"/var/lib/openstack/lib/python3.10/site-packages/sqlalchemy/engine/default.py",
line 736, in do_execute
2025-02-28T10:54:44.153842104Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics cursor.execute(statement, parameters)
2025-02-28T10:54:44.153847123Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics File
"/var/lib/openstack/lib/python3.10/site-packages/pymysql/cursors.py", line 153,
in execute
2025-02-28T10:54:44.153850609Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics result = self._query(query)
2025-02-28T10:54:44.153858566Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics File
"/var/lib/openstack/lib/python3.10/site-packages/pymysql/cursors.py", line 322,
in _query
2025-02-28T10:54:44.153862137Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics conn.query(q)
2025-02-28T10:54:44.153865765Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics File
"/var/lib/openstack/lib/python3.10/site-packages/pymysql/connections.py", line
558, in query
2025-02-28T10:54:44.153869991Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics self._affected_rows =
self._read_query_result(unbuffered=unbuffered)
2025-02-28T10:54:44.153873710Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics File
"/var/lib/openstack/lib/python3.10/site-packages/pymysql/connections.py", line
822, in _read_query_result
2025-02-28T10:54:44.153877267Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics result.read()
2025-02-28T10:54:44.153880644Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics File
"/var/lib/openstack/lib/python3.10/site-packages/pymysql/connections.py", line
1200, in read
2025-02-28T10:54:44.153884719Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics first_packet = self.connection._read_packet()
2025-02-28T10:54:44.153888281Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics File
"/var/lib/openstack/lib/python3.10/site-packages/pymysql/connections.py", line
772, in _read_packet
2025-02-28T10:54:44.153891791Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics packet.raise_for_error()
2025-02-28T10:54:44.153895244Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics File
"/var/lib/openstack/lib/python3.10/site-packages/pymysql/protocol.py", line
221, in raise_for_error
2025-02-28T10:54:44.153898678Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics err.raise_mysql_exception(self._data)
2025-02-28T10:54:44.153902218Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics File
"/var/lib/openstack/lib/python3.10/site-packages/pymysql/err.py", line 143, in
raise_mysql_exception
2025-02-28T10:54:44.153905719Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics raise errorclass(errno, errval)
2025-02-28T10:54:44.153909394Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics pymysql.err.IntegrityError: (1062, "Duplicate entry
'4d9eb917-f05a-4590-a569-42cb59c52bab' for key 'resource_id'")
As I understand it, the culprit is in the 'provideresourceassociation' table.
OVS supports 4 different providers ('single_node', 'ha', dvr', 'dvrha'), and
OVN supports only "ovn" one.
And nothing during the migration (e.g. in
neutron/plugins/ml2/drivers/ovn/db_migration.py) actually changes this table.
Which then leads to both of the errors above.
** Affects: neutron
Importance: Undecided
Assignee: Pavlo Shchelokovskyy (pshchelo)
Status: In Progress
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2103697
Title:
Existing routers become unmanageable after migration from OVS to OVN
Status in neutron:
In Progress
Bug description:
We are investigating the possibilities of migrating from OVS to OVN.
Current state of the migration flow itself we have can be seen here
https://github.com/Mirantis/rockoon/blob/aef29880203481fb6383da09acb5046747f51137/rockoon/cli/ovs_ovn_migration.py
What we found is that after the migration is complete, the pre-
existing routers (created on OVS before migration) become
unmanageable. For example, one can't delete them via neutron API, the
response is HTTP500 with the following error/trace in logs:
31276:2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource [None
req-9e728c4e-349a-4c15-b15b-4cbd0fe84da1 cffbb8f11d5f47d49f5ef63114ca04b6
71b11698a0aa4adb9974f825983b8e1a - - default default] delete failed: No
details.: neutron_lib.callbacks.exceptions.CallbackFailure: Callback
neutron.services.ovn_l3.service_providers.driver_controller.DriverController._clear_router_provider-649091
failed with "'ha'"
31277-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource Traceback
(most recent call last):
31278-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource File
"/var/lib/openstack/lib/python3.10/site-packages/neutron/api/v2/resource.py",
line 98, in resource
31279-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource result =
method(request=request, **args)
31280-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource File
"/var/lib/openstack/lib/python3.10/site-packages/neutron/api/v2/base.py", line
570, in delete
31281-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource return
self._delete(request, id, **kwargs)
31282-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource File
"/var/lib/openstack/lib/python3.10/site-packages/neutron_lib/db/api.py", line
137, in wrapped
31283-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource with
excutils.save_and_reraise_exception():
31284-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource File
"/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line
227, in __exit__
31285-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource
self.force_reraise()
31286-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource File
"/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line
200, in force_reraise
--
31342-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource File
"/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line
227, in __exit__
31343-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource
self.force_reraise()
31344-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource File
"/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line
200, in force_reraise
31345-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource raise
self.value
31346-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource File
"/var/lib/openstack/lib/python3.10/site-packages/neutron_lib/db/utils.py", line
104, in _wrapped
31347-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource return
function(*args, **kwargs)
31348-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource File
"/var/lib/openstack/lib/python3.10/site-packages/neutron_lib/callbacks/manager.py",
line 168, in publish
31349-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource raise
exceptions.CallbackFailure(errors=errors)
31350-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource
neutron_lib.callbacks.exceptions.CallbackFailure: Callback
neutron.services.ovn_l3.service_providers.driver_controller.DriverController._clear_router_provider-649091
failed with "'ha'"
31351-2025-02-25 11:39:26.909 28 ERROR neutron.api.v2.resource
31352:2025-02-25 11:39:26.913 28 INFO neutron.wsgi [None
req-9e728c4e-349a-4c15-b15b-4cbd0fe84da1 cffbb8f11d5f47d49f5ef63114ca04b6
71b11698a0aa4adb9974f825983b8e1a - - default default] 192.168.238.166 "DELETE
/v2.0/routers/bba5be48-2721-4ab2-840a-1cff81148ef2 HTTP/1.1" status: 500 len:
368 time: 5.0718703
Also, after migration one of ovn maintenance tasks (this is Caracal
specific, this code is already removed from master) keeps failing with
025-02-28T10:54:44.153763722Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics [None req-5a85f113-4549-4dca-b768-3e3cc4769f28 - - - - - -]
Failed to call periodic
'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.add_provider_resource_association_to_routers'
(it runs every 5.00 seconds):
neutron_lib.objects.exceptions.NeutronDbObjectDuplicateEntry: Failed to create
a duplicate ProviderResourceAssociation: for attribute(s) ['resource_id'] with
value(s) 4d9eb917-f05a-4590-a569-42cb59c52bab
2025-02-28T10:54:44.153824676Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics Traceback (most recent call last):
2025-02-28T10:54:44.153830586Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics File
"/var/lib/openstack/lib/python3.10/site-packages/sqlalchemy/engine/base.py",
line 1910, in _execute_context
2025-02-28T10:54:44.153834724Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics self.dialect.do_execute(
2025-02-28T10:54:44.153838399Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics File
"/var/lib/openstack/lib/python3.10/site-packages/sqlalchemy/engine/default.py",
line 736, in do_execute
2025-02-28T10:54:44.153842104Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics cursor.execute(statement, parameters)
2025-02-28T10:54:44.153847123Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics File
"/var/lib/openstack/lib/python3.10/site-packages/pymysql/cursors.py", line 153,
in execute
2025-02-28T10:54:44.153850609Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics result = self._query(query)
2025-02-28T10:54:44.153858566Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics File
"/var/lib/openstack/lib/python3.10/site-packages/pymysql/cursors.py", line 322,
in _query
2025-02-28T10:54:44.153862137Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics conn.query(q)
2025-02-28T10:54:44.153865765Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics File
"/var/lib/openstack/lib/python3.10/site-packages/pymysql/connections.py", line
558, in query
2025-02-28T10:54:44.153869991Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics self._affected_rows =
self._read_query_result(unbuffered=unbuffered)
2025-02-28T10:54:44.153873710Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics File
"/var/lib/openstack/lib/python3.10/site-packages/pymysql/connections.py", line
822, in _read_query_result
2025-02-28T10:54:44.153877267Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics result.read()
2025-02-28T10:54:44.153880644Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics File
"/var/lib/openstack/lib/python3.10/site-packages/pymysql/connections.py", line
1200, in read
2025-02-28T10:54:44.153884719Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics first_packet = self.connection._read_packet()
2025-02-28T10:54:44.153888281Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics File
"/var/lib/openstack/lib/python3.10/site-packages/pymysql/connections.py", line
772, in _read_packet
2025-02-28T10:54:44.153891791Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics packet.raise_for_error()
2025-02-28T10:54:44.153895244Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics File
"/var/lib/openstack/lib/python3.10/site-packages/pymysql/protocol.py", line
221, in raise_for_error
2025-02-28T10:54:44.153898678Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics err.raise_mysql_exception(self._data)
2025-02-28T10:54:44.153902218Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics File
"/var/lib/openstack/lib/python3.10/site-packages/pymysql/err.py", line 143, in
raise_mysql_exception
2025-02-28T10:54:44.153905719Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics raise errorclass(errno, errval)
2025-02-28T10:54:44.153909394Z 2025-02-28 10:54:44.151 27 ERROR
futurist.periodics pymysql.err.IntegrityError: (1062, "Duplicate entry
'4d9eb917-f05a-4590-a569-42cb59c52bab' for key 'resource_id'")
As I understand it, the culprit is in the 'provideresourceassociation' table.
OVS supports 4 different providers ('single_node', 'ha', dvr', 'dvrha'), and
OVN supports only "ovn" one.
And nothing during the migration (e.g. in
neutron/plugins/ml2/drivers/ovn/db_migration.py) actually changes this table.
Which then leads to both of the errors above.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2103697/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
