Re: [Openstack-operators] Neutron not clearing up deleted routers

Thu, 08 Jun 2017 10:41:49 -0700 

Ignore that now all,
Managed to fix it by restarting the l3-agent. Looks like it must have been cached in memory. 

Thanks,


On 08/06/17 18:07, Grant Morley wrote:


Hi All,


We have  noticed in our neutron-l3-agent logs that there are a number 
of routers that neutron seems to think exist ( but they no longer 
exist in the data base )  and it is constantly trying to delete them. 
However because they don't appear to exist,  so neutron seems to get 
stuck in a loop of trying to delete routers that are no longer there.



We originally noticed that neutron was trying to delete the 
"qrouter-xxxx-xxx-xxxx" and that didn't exist. So we added the router 
manually with "ip netns add qrouter-xxx-xxx-xxx-xxx" which then 
deletes the router however we then get the following error messages:



2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent [-] Error 
while deleting router da7b633a-233b-46d1-ba3d-315b3bee6a61
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent Traceback 
(most recent call last):
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
"/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/l3/agent.py", 
line 359, in _safe_router_removed
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
self._router_removed(router_id)
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
"/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/l3/agent.py", 
line 377, in _router_removed

2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent ri.delete(self)

2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
"/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", 
line 380, in delete
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
super(HaRouter, self).delete(agent)
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
"/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", 
line 361, in delete
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
self.process_delete(agent)
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
"/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/common/utils.py", 
line 385, in call

2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent self.logger(e)

2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
"/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/oslo_utils/excutils.py", 
line 220, in __exit__
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
self.force_reraise()
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
"/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/oslo_utils/excutils.py", 
line 196, in force_reraise
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
six.reraise(self.type_, self.value, self.tb)
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
"/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/common/utils.py", 
line 382, in call
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent return 
func(*args, **kwargs)
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
"/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", 
line 972, in process_delete
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
self._process_external_on_delete(agent)
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
"/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", 
line 794, in _process_external_on_delete
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
self._process_external_gateway(ex_gw_port, agent.pd)
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
"/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", 
line 693, in _process_external_gateway
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
self.external_gateway_removed(self.ex_gw_port, interface_name)
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
"/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", 
line 371, in external_gateway_removed

2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent interface_name)

2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
"/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", 
line 668, in external_gateway_removed
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
ip_addr['prefixlen']))
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
"/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", 
line 286, in remove_external_gateway_ip
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
device.delete_addr_and_conntrack_state(ip_cidr)
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
"/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", 
line 290, in delete_addr_and_conntrack_state
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
self.addr.delete(cidr)
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
"/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", 
line 580, in delete
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 'dev', 
self.name))
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
"/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", 
line 361, in _as_root
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
use_root_namespace=use_root_namespace)
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
"/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", 
line 94, in _as_root
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
log_fail_as_error=self.log_fail_as_error)
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
"/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", 
line 103, in _execute
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent 
log_fail_as_error=log_fail_as_error)
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent   File 
"/openstack/venvs/neutron-13.3.8/lib/python2.7/site-packages/neutron/agent/linux/utils.py", 
line 140, in execute
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent raise 
RuntimeError(msg)
2017-06-08 16:50:22.340 677 ERROR neutron.agent.l3.agent RuntimeError: 
Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot find device 
"qg-c4080b5c-c9"


Has anyone come across this before?

We don't seem to have an entry for them anywhere from what we can see.

Regards,


--
Grant Morley
Cloud Lead
Absolute DevOps Ltd
Units H, J & K, Gateway 1000, Whittle Way, Stevenage, Herts, SG1 2FP

www.absolutedevops.io <http://www.absolutedevops.io/> 
gr...@absolutedevops.io <mailto:grant@absolutedevops.i> 0845 874 0580



_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


--
Grant Morley
Cloud Lead
Absolute DevOps Ltd
Units H, J & K, Gateway 1000, Whittle Way, Stevenage, Herts, SG1 2FP

www.absolutedevops.io <http://www.absolutedevops.io/> 
gr...@absolutedevops.io <mailto:grant@absolutedevops.i> 0845 874 0580

_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators






















					Reply via email to