[Yahoo-eng-team] [Bug 2025264] [NEW] [ovn][DVR]FIP traffic centralized in DVR environments
Public bug reported: When a port is down, the FIP associated to it get centralized (external_mac removed on NAT table entry) despite DVR being enabled. This also happen when deleting a VM with a FIP associated, where during some period of time, the FIP gets centralized -- time between removing the external_mac from NAT table entry, and the deletion of the NAT table entry. ** Affects: neutron Importance: Undecided Assignee: Luis Tomas Bolivar (ltomasbo) Status: New ** Changed in: neutron Assignee: (unassigned) => Luis Tomas Bolivar (ltomasbo) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2025264 Title: [ovn][DVR]FIP traffic centralized in DVR environments Status in neutron: New Bug description: When a port is down, the FIP associated to it get centralized (external_mac removed on NAT table entry) despite DVR being enabled. This also happen when deleting a VM with a FIP associated, where during some period of time, the FIP gets centralized -- time between removing the external_mac from NAT table entry, and the deletion of the NAT table entry. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2025264/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1932043] Re: nova-ceph-multistore test_resize_server_revert fails with rbd.ReadOnlyImage: [errno 30] RBD read-only image
Seen it today in
https://c83b20527acf2b0f8494-4a0455790e56cb733d68b35ced7c28e7.ssl.cf5.rackcdn.com/886250/2/check/nova-
ceph-multistore/2231918/testr_results.html
** Changed in: nova
Status: Expired => New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1932043
Title:
nova-ceph-multistore test_resize_server_revert fails with
rbd.ReadOnlyImage: [errno 30] RBD read-only image
Status in OpenStack Compute (nova):
New
Bug description:
Traceback (most recent call last):
File "/opt/stack/tempest/tempest/api/compute/base.py", line 222, in
server_check_teardown
waiters.wait_for_server_status(cls.servers_client,
File "/opt/stack/tempest/tempest/common/waiters.py", line 75, in
wait_for_server_status
raise exceptions.BuildErrorException(body['fault'],
tempest.exceptions.BuildErrorException: Server
a1fd599b-909a-4e77-b69d-253b0995bd3d failed to build and is in ERROR status
Details: {'code': 500, 'created': '2021-06-15T14:47:21Z', 'message':
'ReadOnlyImage'}
https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_037/796269/3/check/nova-
ceph-multistore/0373da6/controller/logs/screen-n-cpu.txt
Jun 15 14:47:20.281985 ubuntu-focal-ovh-bhs1-0025123352 nova-compute[114395]:
ERROR nova.compute.manager [None req-ac59ec60-ec07-465d-a82b-ae72be0f2454
tempest-ServerActionsTestJSON-45354220
tempest-ServerActionsTestJSON-45354220-project] [instance:
a1fd599b-909a-4e77-b69d-253b0995bd3d] Setting instance vm_state to ERROR:
rbd.ReadOnlyImage: [errno 30] RBD read-only image (error creating snapshot
b'nova-resize' from b'a1fd599b-909a-4e77-b69d-253b0995bd3d_disk')
Jun 15 14:47:20.281985 ubuntu-focal-ovh-bhs1-0025123352 nova-compute[114395]:
ERROR nova.compute.manager [instance: a1fd599b-909a-4e77-b69d-253b0995bd3d]
Traceback (most recent call last):
Jun 15 14:47:20.281985 ubuntu-focal-ovh-bhs1-0025123352 nova-compute[114395]:
ERROR nova.compute.manager [instance: a1fd599b-909a-4e77-b69d-253b0995bd3d]
File "/opt/stack/nova/nova/compute/manager.py", line 10171, in
_error_out_instance_on_exception
Jun 15 14:47:20.281985 ubuntu-focal-ovh-bhs1-0025123352 nova-compute[114395]:
ERROR nova.compute.manager [instance: a1fd599b-909a-4e77-b69d-253b0995bd3d]
yield
Jun 15 14:47:20.281985 ubuntu-focal-ovh-bhs1-0025123352 nova-compute[114395]:
ERROR nova.compute.manager [instance: a1fd599b-909a-4e77-b69d-253b0995bd3d]
File "/opt/stack/nova/nova/compute/manager.py", line 5862, in
_finish_resize_helper
Jun 15 14:47:20.281985 ubuntu-focal-ovh-bhs1-0025123352 nova-compute[114395]:
ERROR nova.compute.manager [instance: a1fd599b-909a-4e77-b69d-253b0995bd3d]
network_info = self._finish_resize(context, instance, migration,
Jun 15 14:47:20.281985 ubuntu-focal-ovh-bhs1-0025123352 nova-compute[114395]:
ERROR nova.compute.manager [instance: a1fd599b-909a-4e77-b69d-253b0995bd3d]
File "/opt/stack/nova/nova/compute/manager.py", line 5800, in _finish_resize
Jun 15 14:47:20.281985 ubuntu-focal-ovh-bhs1-0025123352 nova-compute[114395]:
ERROR nova.compute.manager [instance: a1fd599b-909a-4e77-b69d-253b0995bd3d]
self._set_instance_info(instance, old_flavor)
Jun 15 14:47:20.281985 ubuntu-focal-ovh-bhs1-0025123352 nova-compute[114395]:
ERROR nova.compute.manager [instance: a1fd599b-909a-4e77-b69d-253b0995bd3d]
File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227,
in __exit__
Jun 15 14:47:20.281985 ubuntu-focal-ovh-bhs1-0025123352 nova-compute[114395]:
ERROR nova.compute.manager [instance: a1fd599b-909a-4e77-b69d-253b0995bd3d]
self.force_reraise()
Jun 15 14:47:20.281985 ubuntu-focal-ovh-bhs1-0025123352 nova-compute[114395]:
ERROR nova.compute.manager [instance: a1fd599b-909a-4e77-b69d-253b0995bd3d]
File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200,
in force_reraise
Jun 15 14:47:20.281985 ubuntu-focal-ovh-bhs1-0025123352 nova-compute[114395]:
ERROR nova.compute.manager [instance: a1fd599b-909a-4e77-b69d-253b0995bd3d]
raise self.value
Jun 15 14:47:20.281985 ubuntu-focal-ovh-bhs1-0025123352 nova-compute[114395]:
ERROR nova.compute.manager [instance: a1fd599b-909a-4e77-b69d-253b0995bd3d]
File "/opt/stack/nova/nova/compute/manager.py", line 5783, in _finish_resize
Jun 15 14:47:20.281985 ubuntu-focal-ovh-bhs1-0025123352 nova-compute[114395]:
ERROR nova.compute.manager [instance: a1fd599b-909a-4e77-b69d-253b0995bd3d]
self.driver.finish_migration(context, migration, instance,
Jun 15 14:47:20.281985 ubuntu-focal-ovh-bhs1-0025123352 nova-compute[114395]:
ERROR nova.compute.manager [instance: a1fd599b-909a-4e77-b69d-253b0995bd3d]
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 10939, in
finish_migration
Jun 15 14:47:20.281985 ubuntu-focal-ovh-bhs1-0025123352 nova-compute[114395]:
ERROR
[Yahoo-eng-team] [Bug 2025246] [NEW] [OVN] Improve log for the exception handling of ovn_l3/plugin.py
Public bug reported:
While debugging an internal issue, the create_router() from the
ovn_l3/plugin.py was raising an exception and as part of the handling of
this exception it was logging an ERROR but, there was no traceback which
makes it really hard to figure out where this error is being raised even
in with debug mode enabled. For example, that's the logs:
['neutron.services.ovn_l3.plugin.RouterAvailabilityZoneMixin._process_az_request-13770945',
'neutron.services.ovn_l3.plugin.OVNL3RouterPlugin.create_router_precommit-181103']
for router, precommit_create _notify_loop
/usr/lib/python3.9/site-packages/neutron_lib/callbacks/manager.py:192
2023-06-26 10:11:39.037 30 DEBUG neutron.db.ovn_revision_numbers_db
[req-18ddc15b-8b4e-4c3c-a360-1d6c10680b39 46d4bcf86b0a4de691d824308920146c
b867c3a5edc8442c946948014a351985 - default default] create_initial_revision
uuid=9969c69e-ca2f-4f6d-bdfd-74cf1febda83, type=routers, rev=-1
create_initial_revision
/usr/lib/python3.9/site-packages/neutron/db/ovn_revision_numbers_db.py:104
2023-06-26 10:11:39.159 30 DEBUG neutron_lib.callbacks.manager
[req-18ddc15b-8b4e-4c3c-a360-1d6c10680b39 46d4bcf86b0a4de691d824308920146c
b867c3a5edc8442c946948014a351985 - default default] Notify callbacks [] for
router, after_create _notify_loop
/usr/lib/python3.9/site-packages/neutron_lib/callbacks/manager.py:192
2023-06-26 10:11:39.160 30 INFO
neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.qos
[req-18ddc15b-8b4e-4c3c-a360-1d6c10680b39 46d4bcf86b0a4de691d824308920146c
b867c3a5edc8442c946948014a351985 - default default] Starting
OVNClientQosExtension
2023-06-26 10:11:39.210 30 ERROR neutron.services.ovn_l3.plugin
[req-18ddc15b-8b4e-4c3c-a360-1d6c10680b39 46d4bcf86b0a4de691d824308920146c
b867c3a5edc8442c946948014a351985 - default default] Unable to create lrouter
for 9969c69e-ca2f-4f6d-bdfd-74cf1febda83:
neutron_lib.exceptions.l3.RouterNotFound: Router
9969c69e-ca2f-4f6d-bdfd-74cf1febda83 could not be found
2023-06-26 10:11:39.219 30 DEBUG neutron_lib.callbacks.manager
[req-18ddc15b-8b4e-4c3c-a360-1d6c10680b39 46d4bcf86b0a4de691d824308920146c
b867c3a5edc8442c946948014a351985 - default default] Notify callbacks [] for
router, before_delete _notify_loop
/usr/lib/python3.9/site-packages/neutron_lib/callbacks/manager.py:192
2023-06-26 10:11:39.386 32 DEBUG neutron.wsgi [-] (32) accepted
('fd00:fd00:fd00:2000::399', 33076, 0, 0) server
/usr/lib/python3.9/site-packages/eventlet/wsgi.py:992
2023-06-26 10:11:39.389 32 INFO neutron.wsgi [-] fd00:fd00:fd00:2000::399 "GET
/ HTTP/1.1" status: 200 len: 244 time: 0.0014989
2023-06-26 10:11:39.406 30 DEBUG neutron_lib.callbacks.manager
[req-18ddc15b-8b4e-4c3c-a360-1d6c10680b39 46d4bcf86b0a4de691d824308920146c
b867c3a5edc8442c946948014a351985 - default default] Notify callbacks [] for
router, precommit_delete _notify_loop
/usr/lib/python3.9/site-packages/neutron_lib/callbacks/manager.py:192
2023-06-26 10:11:39.471 30 DEBUG neutron_lib.callbacks.manager
[req-18ddc15b-8b4e-4c3c-a360-1d6c10680b39 46d4bcf86b0a4de691d824308920146c
b867c3a5edc8442c946948014a351985 - default default] Notify callbacks [] for
router, after_delete _notify_loop
/usr/lib/python3.9/site-packages/neutron_lib/callbacks/manager.py:192
2023-06-26 10:11:39.472 30 DEBUG
neutron.api.rpc.agentnotifiers.l3_rpc_agent_api
[req-18ddc15b-8b4e-4c3c-a360-1d6c10680b39 46d4bcf86b0a4de691d824308920146c
b867c3a5edc8442c946948014a351985 - default default] Fanout notify agent at
l3_agent the message router_deleted on router
9969c69e-ca2f-4f6d-bdfd-74cf1febda83 _notification_fanout
/usr/lib/python3.9/site-packages/neutron/api/rpc/agentnotifiers/l3_rpc_agent_api.py:118
2023-06-26 10:11:39.549 30 INFO neutron.api.v2.resource
[req-18ddc15b-8b4e-4c3c-a360-1d6c10680b39 46d4bcf86b0a4de691d824308920146c
b867c3a5edc8442c946948014a351985 - default default] create failed (client
error): The resource could not be found.
As you can see the error is:
46d4bcf86b0a4de691d824308920146c b867c3a5edc8442c946948014a351985 -
default default] Unable to create lrouter for
9969c69e-ca2f-4f6d-bdfd-74cf1febda83:
neutron_lib.exceptions.l3.RouterNotFound: Router
9969c69e-ca2f-4f6d-bdfd-74cf1febda83 could not be found
But there's no traceback to know where this error has originally being
raised. The create_router() method does a many things and it's very
difficult to figure this out without the traceback.
This LP is about improving the logs for that module in general as I see
other parts where nothing is being logged when things goes bad.
** Affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2025246
Title:
[OVN] Improve log for the exception handling of ovn_l3/plugin.py
Status in neutron:
New
Bug description:
While debugging an internal issue, the create_router() from the
ovn_l3/plugin.py was
