Public bug reported:
It may happen that 2 neutron-server workers will try to remove network
from the DHCP agent in almost the same way. On one of them all will work
fine:
2022-04-19 18:51:26.579 50 WARNING neutron.db.agentschedulers_db
[req-73439ae5-d6a4-432c-92f0-872cf5d19bec - - - - -] Removing network
d2ee3359-3ddb-44ba-ba89-68c8698d550a from agent
1ddb078b-8cfa-48b4-8a87-228f32f7a947 because the agent did not report to
the server in the last 150 seconds.
but on second one it will fail with error:
2022-04-19 18:51:26.634 50 WARNING neutron.db.agentschedulers_db
[req-ee6aab2f-14e8-4080-a1a6-841421cb28e9 - - - - -] Removing network
d2ee3359-3ddb-44ba-ba89-68c8698d550a from agent
1ddb078b-8cfa-48b4-8a87-228f32f7a947 because the agent did not report to the
server in the last 150 seconds.
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db
[req-ee6aab2f-14e8-4080-a1a6-841421cb28e9 - - - - -] Unexpected exception
occurred while removing network d2ee3359-3ddb-44ba-ba89-68c8698d550a from agent
1ddb078b-8cfa-48b4-8a87-228f32f7a947: neutron_lib.exceptions.ObjectNotFound:
Object NetworkDhcpAgentBinding(network_id=d2ee3359-3ddb-44ba-ba89-68c8698d550a,
dhcp_agent_id=1ddb078b-8cfa-48b4-8a87-228f32f7a947) not found.
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db Traceback (most
recent call last):
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db File
"/usr/lib/python3.6/site-packages/neutron/db/agentschedulers_db.py", line 342,
in remove_networks_from_down_agents
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db notify=False)
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db File
"/usr/lib/python3.6/site-packages/neutron/db/agentschedulers_db.py", line 431,
in remove_network_from_dhcp_agent
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db
binding_obj.delete()
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db File
"/usr/lib/python3.6/site-packages/neutron/objects/base.py", line 912, in delete
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db
self._get_composite_keys()))
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db File
"/usr/lib/python3.6/site-packages/neutron/objects/db/api.py", line 94, in
delete_object
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db db_obj =
_safe_get_object(obj_cls, context, **kwargs)
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db File
"/usr/lib/python3.6/site-packages/neutron/objects/db/api.py", line 80, in
_safe_get_object
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db id="%s(%s)"
% (obj_cls.db_model.__name__, key))
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db
neutron_lib.exceptions.ObjectNotFound: Object
NetworkDhcpAgentBinding(network_id=d2ee3359-3ddb-44ba-ba89-68c8698d550a,
dhcp_agent_id=1ddb078b-8cfa-48b4-8a87-228f32f7a947) not found.
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db
** Affects: neutron
Importance: Medium
Assignee: Slawek Kaplonski (slaweq)
Status: In Progress
** Tags: db l3-ipam-dhcp
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1970759
Title:
Race during removal of the network from DHCP agent
Status in neutron:
In Progress
Bug description:
It may happen that 2 neutron-server workers will try to remove network
from the DHCP agent in almost the same way. On one of them all will
work fine:
2022-04-19 18:51:26.579 50 WARNING neutron.db.agentschedulers_db
[req-73439ae5-d6a4-432c-92f0-872cf5d19bec - - - - -] Removing network
d2ee3359-3ddb-44ba-ba89-68c8698d550a from agent
1ddb078b-8cfa-48b4-8a87-228f32f7a947 because the agent did not report
to the server in the last 150 seconds.
but on second one it will fail with error:
2022-04-19 18:51:26.634 50 WARNING neutron.db.agentschedulers_db
[req-ee6aab2f-14e8-4080-a1a6-841421cb28e9 - - - - -] Removing network
d2ee3359-3ddb-44ba-ba89-68c8698d550a from agent
1ddb078b-8cfa-48b4-8a87-228f32f7a947 because the agent did not report to the
server in the last 150 seconds.
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db
[req-ee6aab2f-14e8-4080-a1a6-841421cb28e9 - - - - -] Unexpected exception
occurred while removing network d2ee3359-3ddb-44ba-ba89-68c8698d550a from agent
1ddb078b-8cfa-48b4-8a87-228f32f7a947: neutron_lib.exceptions.ObjectNotFound:
Object NetworkDhcpAgentBinding(network_id=d2ee3359-3ddb-44ba-ba89-68c8698d550a,
dhcp_agent_id=1ddb078b-8cfa-48b4-8a87-228f32f7a947) not found.
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db Traceback
(most recent call last):
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db File
"/usr/lib/python3.6/site-packages/neutron/db/agentschedulers_db.py", line 342,
in remove_networks_from_down_agents
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db
notify=False)
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db File
"/usr/lib/python3.6/site-packages/neutron/db/agentschedulers_db.py", line 431,
in remove_network_from_dhcp_agent
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db
binding_obj.delete()
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db File
"/usr/lib/python3.6/site-packages/neutron/objects/base.py", line 912, in delete
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db
self._get_composite_keys()))
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db File
"/usr/lib/python3.6/site-packages/neutron/objects/db/api.py", line 94, in
delete_object
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db db_obj =
_safe_get_object(obj_cls, context, **kwargs)
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db File
"/usr/lib/python3.6/site-packages/neutron/objects/db/api.py", line 80, in
_safe_get_object
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db
id="%s(%s)" % (obj_cls.db_model.__name__, key))
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db
neutron_lib.exceptions.ObjectNotFound: Object
NetworkDhcpAgentBinding(network_id=d2ee3359-3ddb-44ba-ba89-68c8698d550a,
dhcp_agent_id=1ddb078b-8cfa-48b4-8a87-228f32f7a947) not found.
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1970759/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
