@racb done. ** Description changed:
[Impact] neutron-l3-agent restart causes partial loss of fip information such that fip removal from vm results in ip rules left behind which breaks external network access for that vm. [Test Case] * deploy openstack with dvr enabled * create distributed router, network etc * create a vm and attach a floating ip * go to compute host on which vm is running and restart neutron-l3-agent * tail -f /var/log/neutron/neutron-l3-agent.log until it settles * remove fip from vm * run https://gist.github.com/dosaboy/eca8dcd4560f68d856f465ca8382c58b on that compute node * should return with "nothing to do" [Regression Potential] - none expected + + the patch is reloading, on agent startup, information associated with + floating ips, specifically the information needed to delete ip rules and + rule priorities associated with a floating ip. Since that is essentially + read-only I don't envisage a regression potential. When the l3-agent + comes to use that information to delete the floating ip an error could + occur if the information it is trying to delete no longer exists but + that would not be a problem introduced by this patch so again, I don't + envisage any potential for regressions from this patch since it doesn't + change behavior in any way other than allowing the l3-agent to behave + the same as if it hadn't been restarted. [Other Info] patched neutron l3 agent will reload info for *used* floating ips when restarted BUT if there are ip rules left behind from fips removed prior to using a pathed neutron then manual cleanup is still required and for that you can use https://gist.github.com/dosaboy/eca8dcd4560f68d856f465ca8382c58b. - + -------------------------------------------------------------------------- With Bionic Stein using dvr_snat if I add a floating ip to a vm then remove the floating ip, the corresponding ip rules in the associated qrouter ns local to the instance are not deleted which results in no longer being able to reach the external network because packets are still sent to the fip namespace (via rfp-/fpr-) e.g. in my compute host running a vm whose address is 192.168.21.28 for which i have removed the fip I still see: # ip netns exec qrouter-5e45608f-33d4-41bf-b3ba-915adf612e65 ip rule list 0: from all lookup local 32765: from 192.168.21.28 lookup 16 32766: from all lookup main 32767: from all lookup default 3232240897: from 192.168.21.1/24 lookup 3232240897 3232241231: from 192.168.22.79/24 lookup 3232241231 And table 16 leads to: # ip netns exec qrouter-5e45608f-33d4-41bf-b3ba-915adf612e65 ip route show table 16 default via 169.254.109.249 dev rfp-5e45608f-3 Which results in the instance no longer being able to reach the external network (packets are never sent to the snat- ns in my case). The workaround is to delete that ip rule but neutron should be taking care of this. Looks like the culprit is in neutron/agent/l3/dvr_local_router.py:floating_ip_removed_dist Note that the NAT rules were successfully removed from iptables so looks like it is just this bit that is left behind. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1891673 Title: qrouter ns ip rules not deleted when fip removed from vm To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1891673/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
