Public bug reported:
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent [-] Failed to process
compatible router: edf1cc99-879a-4fe5-a7b2-d19acb8fdcbf:
neutron.privileged.agent.linux.ip_lib.InterfaceOperationNotSupported: Operation
not supported on interface qg-4bb9d20b-a0, namespace
qrouter-edf1cc99-879a-4fe5-a7b2-d19acb8fdcbf.
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent Traceback (most recent
call last):
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/agent.py",
line 848, in _process_routers_if_compatible
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent
self._process_router_if_compatible(router)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/agent.py",
line 639, in _process_router_if_compatible
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent
self._process_updated_router(router)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/agent.py",
line 719, in _process_updated_router
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent ri.process()
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/common/utils.py", line
177, in call
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent self.logger(e)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_utils/excutils.py", line
227, in __exit__
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent self.force_reraise()
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_utils/excutils.py", line
200, in force_reraise
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent raise self.value
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/common/utils.py", line
174, in call
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent return func(*args,
**kwargs)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/router_info.py",
line 1312, in process
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent
self.process_external()
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/decorator.py", line 232, in fun
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent return caller(func,
*(extras + args), **kw)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/common/coordination.py",
line 78, in _synchronized
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent return f(*a, **k)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/router_info.py",
line 1046, in process_external
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent
self._process_external_gateway(ex_gw_port)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/router_info.py",
line 931, in _process_external_gateway
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent
self.external_gateway_updated(ex_gw_port, interface_name)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/router_info.py",
line 884, in external_gateway_updated
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent
self._external_gateway_added(
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/router_info.py",
line 822, in _external_gateway_added
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent
self._external_gateway_settings(ex_gw_port, interface_name,
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/router_info.py",
line 835, in _external_gateway_settings
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent
self.driver.init_router_port(
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/linux/interface.py",
line 171, in init_router_port
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent
self.init_l3(device_name=device_name,
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/linux/interface.py",
line 146, in init_l3
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent
device.delete_addr_and_conntrack_state(ip_cidr)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/linux/ip_lib.py",
line 361, in delete_addr_and_conntrack_state
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent
self.addr.delete(cidr)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/linux/ip_lib.py",
line 540, in delete
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent
delete_ip_address(cidr, self.name, self._parent.namespace)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/linux/ip_lib.py",
line 834, in delete_ip_address
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent
privileged.delete_ip_address(
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_privsep/priv_context.py",
line 271, in _wrap
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent return
self.channel.remote_call(name, args, kwargs,
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_privsep/daemon.py", line
215, in remote_call
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent raise
exc_type(*result[2])
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent
neutron.privileged.agent.linux.ip_lib.InterfaceOperationNotSupported: Operation
not supported on interface qg-4bb9d20b-a0, namespace
qrouter-edf1cc99-879a-4fe5-a7b2-d19acb8fdcbf.
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent
2022-10-03 03:23:33.869 21 WARNING neutron.agent.l3.agent [-] Hit retry limit
with router update for edf1cc99-879a-4fe5-a7b2-d19acb8fdcbf, action 3
version info
yoga deoployed with kolla-ansible using debian ubuntu or centos image and popos
22.04 host
popos 22.04 is based on ubuntu 22.04 and uses nftables intead of legacy
ip tables
in the contianer i noticed that iptable -L is slow but iptable -S is fast
it is not clare if this is a race or some other issue but intially the router
is fine and hte floating ips work and restarting the l3 agent fixes the issue
for a time then it breaks with this message.
** Affects: neutron
Importance: Undecided
Status: New
** Tags: l3 ovs
** Tags added: l3 ovs
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1991501
Title:
l3 agent fails to init router breaking floating ips
Status in neutron:
New
Bug description:
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent [-] Failed to process
compatible router: edf1cc99-879a-4fe5-a7b2-d19acb8fdcbf:
neutron.privileged.agent.linux.ip_lib.InterfaceOperationNotSupported: Operation
not supported on interface qg-4bb9d20b-a0, namespace
qrouter-edf1cc99-879a-4fe5-a7b2-d19acb8fdcbf.
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent Traceback (most
recent call last):
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/agent.py",
line 848, in _process_routers_if_compatible
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent
self._process_router_if_compatible(router)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/agent.py",
line 639, in _process_router_if_compatible
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent
self._process_updated_router(router)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/agent.py",
line 719, in _process_updated_router
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent ri.process()
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/common/utils.py", line
177, in call
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent self.logger(e)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_utils/excutils.py", line
227, in __exit__
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent
self.force_reraise()
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_utils/excutils.py", line
200, in force_reraise
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent raise self.value
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/common/utils.py", line
174, in call
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent return
func(*args, **kwargs)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/router_info.py",
line 1312, in process
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent
self.process_external()
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/decorator.py", line 232, in fun
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent return
caller(func, *(extras + args), **kw)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/common/coordination.py",
line 78, in _synchronized
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent return f(*a, **k)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/router_info.py",
line 1046, in process_external
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent
self._process_external_gateway(ex_gw_port)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/router_info.py",
line 931, in _process_external_gateway
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent
self.external_gateway_updated(ex_gw_port, interface_name)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/router_info.py",
line 884, in external_gateway_updated
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent
self._external_gateway_added(
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/router_info.py",
line 822, in _external_gateway_added
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent
self._external_gateway_settings(ex_gw_port, interface_name,
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/router_info.py",
line 835, in _external_gateway_settings
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent
self.driver.init_router_port(
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/linux/interface.py",
line 171, in init_router_port
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent
self.init_l3(device_name=device_name,
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/linux/interface.py",
line 146, in init_l3
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent
device.delete_addr_and_conntrack_state(ip_cidr)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/linux/ip_lib.py",
line 361, in delete_addr_and_conntrack_state
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent
self.addr.delete(cidr)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/linux/ip_lib.py",
line 540, in delete
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent
delete_ip_address(cidr, self.name, self._parent.namespace)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/linux/ip_lib.py",
line 834, in delete_ip_address
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent
privileged.delete_ip_address(
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_privsep/priv_context.py",
line 271, in _wrap
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent return
self.channel.remote_call(name, args, kwargs,
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File
"/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_privsep/daemon.py", line
215, in remote_call
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent raise
exc_type(*result[2])
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent
neutron.privileged.agent.linux.ip_lib.InterfaceOperationNotSupported: Operation
not supported on interface qg-4bb9d20b-a0, namespace
qrouter-edf1cc99-879a-4fe5-a7b2-d19acb8fdcbf.
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent
2022-10-03 03:23:33.869 21 WARNING neutron.agent.l3.agent [-] Hit retry limit
with router update for edf1cc99-879a-4fe5-a7b2-d19acb8fdcbf, action 3
version info
yoga deoployed with kolla-ansible using debian ubuntu or centos image and
popos 22.04 host
popos 22.04 is based on ubuntu 22.04 and uses nftables intead of
legacy ip tables
in the contianer i noticed that iptable -L is slow but iptable -S is fast
it is not clare if this is a race or some other issue but intially the router
is fine and hte floating ips work and restarting the l3 agent fixes the issue
for a time then it breaks with this message.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1991501/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
