Public bug reported: If a fip ip is removed of a network with a distributed router:
openstack server remove floating ip X The conntrack rules aren't deleted in the qrouter and the qrouter continues doing nating of the ongoing connections. overcloud) [stack@undercloud-0 ~]$ openstack router show router +-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | nova | | created_at | 2019-02-20T15:46:53Z | | description | | | distributed | True | | external_gateway_info | {"network_id": "15a5c01e-4e42-4890-a850-db4f97bb5834", "enable_snat": true, "external_fixed_ips": [{"subnet_id": "c59ae813-1df7-4a14-9eba-be2e35afa13e", "ip_address": "10.0.0.214"}]} | | flavor_id | None | | ha | False | | id | d01c89b0-c2df-46e2-9c12-8d14b1c8ce9a | | interfaces_info | [{"subnet_id": "5e8ddfa7-d546-4f59-94d1-e2b65e8ecdb6", "ip_address": "10.2.0.8", "port_id": "06c6e9d3-2c6b-40b8-8919-92be6efd0153"}, {"subnet_id": "5e8ddfa7-d546-4f59-94d1-e2b65e8ecdb6", "ip_address": "10.2.0.1", "port_id": "c47b0417-7dbe-4434-8c50-72a78e6335a1"}] | | name | router | | project_id | 9447276fedbf4c4eab15494f8d187d97 | | revision_number | 13 | | routes | | | status | ACTIVE | | tags | | | updated_at | 2019-03-05T11:31:34Z | +-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ (overcloud) [stack@undercloud-0 ~]$ (overcloud) [stack@undercloud-0 ~]$ (overcloud) [stack@undercloud-0 ~]$ openstack server list +--------------------------------------+------------------------+--------+------------------------------------+--------+--------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+------------------------+--------+------------------------------------+--------+--------+ | 8aad2992-b068-4378-83f8-965b59d04d8d | selfservice2-instance4 | ACTIVE | selfservice2=10.2.0.12, 10.0.0.210 | cirros | cirros | | 028e0696-e666-4c09-802a-49a126a6346d | selfservice2-instance3 | ACTIVE | selfservice2=10.2.0.27 | cirros | cirros | | 46432868-4d11-4c9d-a910-ddf246c78378 | selfservice2-instance2 | ACTIVE | selfservice2=10.2.0.14 | cirros | cirros | | fca6541e-d846-45fd-8970-8ba27ff708d6 | selfservice2-instance1 | ACTIVE | selfservice2=10.2.0.30 | cirros | cirros | +--------------------------------------+------------------------+--------+------------------------------------+--------+--------+ $ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast qlen 1000 link/ether fa:16:3e:6e:0e:f7 brd ff:ff:ff:ff:ff:ff inet 10.2.0.12/24 brd 10.2.0.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe6e:ef7/64 scope link valid_lft forever preferred_lft forever $ hostname selfservice2-instance4 $ $ $ ping 8.8.8.8 PING 8.8.8.8 (8.8.8.8): 56 data bytes 64 bytes from 8.8.8.8: seq=0 ttl=116 time=49.554 ms 64 bytes from 8.8.8.8: seq=1 ttl=116 time=48.893 ms 64 bytes from 8.8.8.8: seq=2 ttl=116 time=48.897 ms 64 bytes from 8.8.8.8: seq=3 ttl=116 time=48.988 ms .... [heat-admin@compute-1 ~]$ sudo su [root@compute-1 heat-admin]# yum install conntrack-tools .... [root@compute-1 heat-admin]# conntrack -L icmp 1 29 src=10.2.0.12 dst=8.8.8.8 type=8 code=0 id=45825 src=8.8.8.8 dst=10.0.0.210 type=0 code=0 id=45825 mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=1 conntrack v1.4.4 (conntrack-tools): 1 flow entries have been shown. [root@compute-1 heat-admin]# (overcloud) [stack@undercloud-0 ~]$ openstack server remove floating ip selfservice2-instance4 10.0.0.210 (overcloud) [stack@undercloud-0 ~]$ date Wed Mar 6 04:59:40 EST 2019 (overcloud) [stack@undercloud-0 ~]$ Then connectivity is lost: 64 bytes from 8.8.8.8: seq=813 ttl=116 time=48.753 ms 64 bytes from 8.8.8.8: seq=814 ttl=116 time=48.895 ms because conntrack flow is not deleted: [root@compute-1 heat-admin]# conntrack -L icmp 1 29 src=10.2.0.12 dst=8.8.8.8 type=8 code=0 id=45825 src=8.8.8.8 dst=10.0.0.210 type=0 code=0 id=45825 mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=1 conntrack v1.4.4 (conntrack-tools): 1 flow entries have been shown. [root@compute-1 heat-admin]# date Wed Mar 6 10:02:04 UTC 2019 [root@compute-1 heat-admin]# conntrack -L icmp 1 29 src=10.2.0.12 dst=8.8.8.8 type=8 code=0 id=45825 src=8.8.8.8 dst=10.0.0.210 type=0 code=0 id=45825 mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=1 conntrack v1.4.4 (conntrack-tools): 1 flow entries have been shown. [root@compute-1 heat-admin]# If I delete the flow the connectiviy is recovered(same behavior that without dvr): [root@compute-1 heat-admin]# conntrack -D icmp 1 29 src=10.2.0.12 dst=8.8.8.8 type=8 code=0 id=45825 src=8.8.8.8 dst=10.0.0.210 type=0 code=0 id=45825 mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=1 conntrack v1.4.4 (conntrack-tools): 1 flow entries have been deleted. [root@compute-1 heat-admin]# conntrack -L icmp 1 28 src=10.2.0.12 dst=8.8.8.8 type=8 code=0 id=45825 [UNREPLIED] src=8.8.8.8 dst=10.2.0.12 type=0 code=0 id=45825 mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=1 conntrack v1.4.4 (conntrack-tools): 1 flow entries have been shown. [root@compute-1 heat-admin]# .... 64 bytes from 8.8.8.8: seq=812 ttl=116 time=48.791 ms 64 bytes from 8.8.8.8: seq=813 ttl=116 time=48.753 ms 64 bytes from 8.8.8.8: seq=814 ttl=116 time=48.895 ms 64 bytes from 8.8.8.8: seq=980 ttl=117 time=49.979 ms 64 bytes from 8.8.8.8: seq=981 ttl=117 time=49.164 ms 64 bytes from 8.8.8.8: seq=982 ttl=117 time=49.524 ms 64 bytes from 8.8.8.8: seq=983 ttl=117 time=49.143 ms .... ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1818805 Title: Conntrack rules in the qrouter are not deleted when a fip is removed with dvr Status in neutron: New Bug description: If a fip ip is removed of a network with a distributed router: openstack server remove floating ip X The conntrack rules aren't deleted in the qrouter and the qrouter continues doing nating of the ongoing connections. overcloud) [stack@undercloud-0 ~]$ openstack router show router +-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | nova | | created_at | 2019-02-20T15:46:53Z | | description | | | distributed | True | | external_gateway_info | {"network_id": "15a5c01e-4e42-4890-a850-db4f97bb5834", "enable_snat": true, "external_fixed_ips": [{"subnet_id": "c59ae813-1df7-4a14-9eba-be2e35afa13e", "ip_address": "10.0.0.214"}]} | | flavor_id | None | | ha | False | | id | d01c89b0-c2df-46e2-9c12-8d14b1c8ce9a | | interfaces_info | [{"subnet_id": "5e8ddfa7-d546-4f59-94d1-e2b65e8ecdb6", "ip_address": "10.2.0.8", "port_id": "06c6e9d3-2c6b-40b8-8919-92be6efd0153"}, {"subnet_id": "5e8ddfa7-d546-4f59-94d1-e2b65e8ecdb6", "ip_address": "10.2.0.1", "port_id": "c47b0417-7dbe-4434-8c50-72a78e6335a1"}] | | name | router | | project_id | 9447276fedbf4c4eab15494f8d187d97 | | revision_number | 13 | | routes | | | status | ACTIVE | | tags | | | updated_at | 2019-03-05T11:31:34Z | +-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ (overcloud) [stack@undercloud-0 ~]$ (overcloud) [stack@undercloud-0 ~]$ (overcloud) [stack@undercloud-0 ~]$ openstack server list +--------------------------------------+------------------------+--------+------------------------------------+--------+--------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+------------------------+--------+------------------------------------+--------+--------+ | 8aad2992-b068-4378-83f8-965b59d04d8d | selfservice2-instance4 | ACTIVE | selfservice2=10.2.0.12, 10.0.0.210 | cirros | cirros | | 028e0696-e666-4c09-802a-49a126a6346d | selfservice2-instance3 | ACTIVE | selfservice2=10.2.0.27 | cirros | cirros | | 46432868-4d11-4c9d-a910-ddf246c78378 | selfservice2-instance2 | ACTIVE | selfservice2=10.2.0.14 | cirros | cirros | | fca6541e-d846-45fd-8970-8ba27ff708d6 | selfservice2-instance1 | ACTIVE | selfservice2=10.2.0.30 | cirros | cirros | +--------------------------------------+------------------------+--------+------------------------------------+--------+--------+ $ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast qlen 1000 link/ether fa:16:3e:6e:0e:f7 brd ff:ff:ff:ff:ff:ff inet 10.2.0.12/24 brd 10.2.0.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe6e:ef7/64 scope link valid_lft forever preferred_lft forever $ hostname selfservice2-instance4 $ $ $ ping 8.8.8.8 PING 8.8.8.8 (8.8.8.8): 56 data bytes 64 bytes from 8.8.8.8: seq=0 ttl=116 time=49.554 ms 64 bytes from 8.8.8.8: seq=1 ttl=116 time=48.893 ms 64 bytes from 8.8.8.8: seq=2 ttl=116 time=48.897 ms 64 bytes from 8.8.8.8: seq=3 ttl=116 time=48.988 ms .... [heat-admin@compute-1 ~]$ sudo su [root@compute-1 heat-admin]# yum install conntrack-tools .... [root@compute-1 heat-admin]# conntrack -L icmp 1 29 src=10.2.0.12 dst=8.8.8.8 type=8 code=0 id=45825 src=8.8.8.8 dst=10.0.0.210 type=0 code=0 id=45825 mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=1 conntrack v1.4.4 (conntrack-tools): 1 flow entries have been shown. [root@compute-1 heat-admin]# (overcloud) [stack@undercloud-0 ~]$ openstack server remove floating ip selfservice2-instance4 10.0.0.210 (overcloud) [stack@undercloud-0 ~]$ date Wed Mar 6 04:59:40 EST 2019 (overcloud) [stack@undercloud-0 ~]$ Then connectivity is lost: 64 bytes from 8.8.8.8: seq=813 ttl=116 time=48.753 ms 64 bytes from 8.8.8.8: seq=814 ttl=116 time=48.895 ms because conntrack flow is not deleted: [root@compute-1 heat-admin]# conntrack -L icmp 1 29 src=10.2.0.12 dst=8.8.8.8 type=8 code=0 id=45825 src=8.8.8.8 dst=10.0.0.210 type=0 code=0 id=45825 mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=1 conntrack v1.4.4 (conntrack-tools): 1 flow entries have been shown. [root@compute-1 heat-admin]# date Wed Mar 6 10:02:04 UTC 2019 [root@compute-1 heat-admin]# conntrack -L icmp 1 29 src=10.2.0.12 dst=8.8.8.8 type=8 code=0 id=45825 src=8.8.8.8 dst=10.0.0.210 type=0 code=0 id=45825 mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=1 conntrack v1.4.4 (conntrack-tools): 1 flow entries have been shown. [root@compute-1 heat-admin]# If I delete the flow the connectiviy is recovered(same behavior that without dvr): [root@compute-1 heat-admin]# conntrack -D icmp 1 29 src=10.2.0.12 dst=8.8.8.8 type=8 code=0 id=45825 src=8.8.8.8 dst=10.0.0.210 type=0 code=0 id=45825 mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=1 conntrack v1.4.4 (conntrack-tools): 1 flow entries have been deleted. [root@compute-1 heat-admin]# conntrack -L icmp 1 28 src=10.2.0.12 dst=8.8.8.8 type=8 code=0 id=45825 [UNREPLIED] src=8.8.8.8 dst=10.2.0.12 type=0 code=0 id=45825 mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=1 conntrack v1.4.4 (conntrack-tools): 1 flow entries have been shown. [root@compute-1 heat-admin]# .... 64 bytes from 8.8.8.8: seq=812 ttl=116 time=48.791 ms 64 bytes from 8.8.8.8: seq=813 ttl=116 time=48.753 ms 64 bytes from 8.8.8.8: seq=814 ttl=116 time=48.895 ms 64 bytes from 8.8.8.8: seq=980 ttl=117 time=49.979 ms 64 bytes from 8.8.8.8: seq=981 ttl=117 time=49.164 ms 64 bytes from 8.8.8.8: seq=982 ttl=117 time=49.524 ms 64 bytes from 8.8.8.8: seq=983 ttl=117 time=49.143 ms .... To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1818805/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp