Public bug reported: I run into a problem when neutron dhcp-agent is still replying to the old address confirmation. Simple steps to reproduce: - create a port with IPv6 address in dhcpv6-stateful subnet - create a VM with cloud-init inside - change the IPv6 port address - reboot the VM
Here are my commands: $ openstack subnet create --subnet-range 2001:db8:123::/64 --ip-version 6 --ipv6-address-mode dhcpv6-stateful --network public subv6 $ openstack subnet list --network public +--------------------------------------+-------+--------------------------------------+-------------------+ | ID | Name | Network | Subnet | +--------------------------------------+-------+--------------------------------------+-------------------+ | 6d9a7fb5-5c1b-4759-b32b-5720b5cedbf4 | subv4 | f1f3d967-26db-41b3-b6f6-1d5356e33a84 | 10.136.16.0/22 | | 76db898c-6a7a-4301-9253-23241cafaa83 | subv6 | f1f3d967-26db-41b3-b6f6-1d5356e33a84 | 2001:db8:123::/64 | +--------------------------------------+-------+--------------------------------------+-------------------+ $ $ openstack port create my-port --network public --fixed-ip ip-address=10.136.17.163 --fixed-ip ip-address=2001:db8:123::111 $ openstack server create test --flavor m1.small --port my-port --image CentOS-7-x86_64-GenericCloud-2009.qcow2 --key-name key --use-config-drive Check IPv6 address inside VM (it's correct): [centos@test ~]$ ip a s eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether fa:16:3e:2e:66:ac brd ff:ff:ff:ff:ff:ff inet 10.136.17.163/22 brd 10.136.19.255 scope global dynamic eth0 valid_lft 86371sec preferred_lft 86371sec inet6 2001:db8:123::111/128 scope global dynamic valid_lft 7473sec preferred_lft 7173sec inet6 fe80::f816:3eff:fe2e:66ac/64 scope link valid_lft forever preferred_lft forever [centos@test ~]$ Change IPv6 address and reboot the VM: $ openstack port set my-port --no-fixed-ip --fixed-ip ip-address=10.136.17.163 --fixed-ip ip-address=2001:db8:123::222 $ openstack server reboot test [centos@test ~]$ ip a s eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether fa:16:3e:2e:66:ac brd ff:ff:ff:ff:ff:ff inet 10.136.17.163/22 brd 10.136.19.255 scope global dynamic eth0 valid_lft 86382sec preferred_lft 86382sec inet6 2001:db8:123::111/128 scope global dynamic valid_lft 7482sec preferred_lft 7182sec inet6 fe80::f816:3eff:fe2e:66ac/64 scope link valid_lft forever preferred_lft forever [centos@test ~]$ ^^ you can see the VM got the old IPv6 address and actually all traffic is blocked by port-security feature. If I remove a lease file and re- spawn a dhclient, all is fine: [centos@test ~]$ ps axf | grep dhcl 780 ? Ss 0:00 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient--eth0.lease -pf /var/run/dhclient-eth0.pid -H test eth0 868 ? Ss 0:00 /sbin/dhclient -6 -1 -lf /var/lib/dhclient/dhclient6--eth0.lease -pf /var/run/dhclient6-eth0.pid eth0 -H test 1371 pts/0 S+ 0:00 \_ grep --color=auto dhcl [centos@test ~]$ sudo kill -9 868 [centos@test ~]$ sudo ip addr del 2001:db8:123::111/128 dev eth0 [centos@test ~]$ sudo rm -rf /var/lib/dhclient/dhclient6--eth0.lease [centos@test ~]$ sudo /sbin/dhclient -6 -1 -lf /var/lib/dhclient/dhclient6--eth0.lease -pf /var/run/dhclient6-eth0.pid eth0 -H test [centos@test ~]$ ip a s eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether fa:16:3e:2e:66:ac brd ff:ff:ff:ff:ff:ff inet 10.136.17.163/22 brd 10.136.19.255 scope global dynamic eth0 valid_lft 86319sec preferred_lft 86319sec inet6 2001:db8:123::222/128 scope global dynamic valid_lft 7481sec preferred_lft 7181sec inet6 fe80::f816:3eff:fe2e:66ac/64 scope link valid_lft forever preferred_lft forever [centos@test ~]$ I found some logic with dhcpv6 leases removing here: https://opendev.org/openstack/neutron/src/commit/e7b70521d0e230143a80974e7e4795a2acafcc9b/neutron/agent/linux/dhcp.py#L600 but it looks like it doesn't help in case of DHCPCONFIRM client request: In the dnsmasq logs I see the following DHCPCONFIRM->DHCPREPLY messages exchange after the VM came back after the reboot (see also https://datatracker.ietf.org/doc/html/rfc3315#page-50): Feb 1 16:49:12 dnsmasq-dhcp[1360521]: DHCPREQUEST(tapc233cb5c-8f) 10.136.17.163 fa:16:3e:2e:66:ac Feb 1 16:49:12 dnsmasq-dhcp[1360521]: DHCPACK(tapc233cb5c-8f) 10.136.17.163 fa:16:3e:2e:66:ac host-10-136-17-163 Feb 1 16:49:15 dnsmasq-dhcp[1360521]: DHCPCONFIRM(tapc233cb5c-8f) 00:01:00:01:29:8c:20:5e:fa:16:3e:2e:66:ac Feb 1 16:49:15 dnsmasq-dhcp[1360521]: DHCPREPLY(tapc233cb5c-8f) 2001:db8:123::111 00:01:00:01:29:8c:20:5e:fa:16:3e:2e:66:ac host-2001-db8-123--222 ** Affects: neutron Importance: Undecided Status: New ** Description changed: I run into a problem when neutron dhcp-agent is still replying to the old address confirmation. Simple steps to reproduce: - create a port with IPv6 address in dhcpv6-stateful subnet - create a VM with cloud-init inside - change the IPv6 port address - reboot the VM Here are my commands: $ openstack subnet create --subnet-range 2001:db8:123::/64 --ip-version 6 --ipv6-address-mode dhcpv6-stateful --network public subv6 $ openstack subnet list --network public +--------------------------------------+-------+--------------------------------------+-------------------+ | ID | Name | Network | Subnet | +--------------------------------------+-------+--------------------------------------+-------------------+ | 6d9a7fb5-5c1b-4759-b32b-5720b5cedbf4 | subv4 | f1f3d967-26db-41b3-b6f6-1d5356e33a84 | 10.136.16.0/22 | | 76db898c-6a7a-4301-9253-23241cafaa83 | subv6 | f1f3d967-26db-41b3-b6f6-1d5356e33a84 | 2001:db8:123::/64 | +--------------------------------------+-------+--------------------------------------+-------------------+ $ $ openstack port create my-port --network public --fixed-ip ip-address=10.136.17.163 --fixed-ip ip-address=2001:db8:123::111 $ openstack server create test --flavor m1.small --port my-port --image CentOS-7-x86_64-GenericCloud-2009.qcow2 --key-name key --use-config-drive Check IPv6 address inside VM (it's correct): [centos@test ~]$ ip a s eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 - link/ether fa:16:3e:2e:66:ac brd ff:ff:ff:ff:ff:ff - inet 10.136.17.163/22 brd 10.136.19.255 scope global dynamic eth0 - valid_lft 86371sec preferred_lft 86371sec - inet6 2001:db8:123::111/128 scope global dynamic - valid_lft 7473sec preferred_lft 7173sec - inet6 fe80::f816:3eff:fe2e:66ac/64 scope link - valid_lft forever preferred_lft forever + link/ether fa:16:3e:2e:66:ac brd ff:ff:ff:ff:ff:ff + inet 10.136.17.163/22 brd 10.136.19.255 scope global dynamic eth0 + valid_lft 86371sec preferred_lft 86371sec + inet6 2001:db8:123::111/128 scope global dynamic + valid_lft 7473sec preferred_lft 7173sec + inet6 fe80::f816:3eff:fe2e:66ac/64 scope link + valid_lft forever preferred_lft forever [centos@test ~]$ Change IPv6 address and reboot the VM: $ openstack port set my-port --no-fixed-ip --fixed-ip ip-address=10.136.17.163 --fixed-ip ip-address=2001:db8:123::222 $ openstack server reboot test [centos@test ~]$ ip a s eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 - link/ether fa:16:3e:2e:66:ac brd ff:ff:ff:ff:ff:ff - inet 10.136.17.163/22 brd 10.136.19.255 scope global dynamic eth0 - valid_lft 86382sec preferred_lft 86382sec - inet6 2001:db8:123::111/128 scope global dynamic - valid_lft 7482sec preferred_lft 7182sec - inet6 fe80::f816:3eff:fe2e:66ac/64 scope link - valid_lft forever preferred_lft forever + link/ether fa:16:3e:2e:66:ac brd ff:ff:ff:ff:ff:ff + inet 10.136.17.163/22 brd 10.136.19.255 scope global dynamic eth0 + valid_lft 86382sec preferred_lft 86382sec + inet6 2001:db8:123::111/128 scope global dynamic + valid_lft 7482sec preferred_lft 7182sec + inet6 fe80::f816:3eff:fe2e:66ac/64 scope link + valid_lft forever preferred_lft forever [centos@test ~]$ ^^ you can see the VM got the old IPv6 address and actually all traffic is blocked by port-security feature. If I remove a lease file and re- spawn a dhclient, all is fine: [centos@test ~]$ ps axf | grep dhcl - 780 ? Ss 0:00 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient--eth0.lease -pf /var/run/dhclient-eth0.pid -H test eth0 - 868 ? Ss 0:00 /sbin/dhclient -6 -1 -lf /var/lib/dhclient/dhclient6--eth0.lease -pf /var/run/dhclient6-eth0.pid eth0 -H test - 1371 pts/0 S+ 0:00 \_ grep --color=auto dhcl + 780 ? Ss 0:00 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient--eth0.lease -pf /var/run/dhclient-eth0.pid -H test eth0 + 868 ? Ss 0:00 /sbin/dhclient -6 -1 -lf /var/lib/dhclient/dhclient6--eth0.lease -pf /var/run/dhclient6-eth0.pid eth0 -H test + 1371 pts/0 S+ 0:00 \_ grep --color=auto dhcl [centos@test ~]$ sudo kill -9 868 [centos@test ~]$ sudo ip addr del 2001:db8:123::111/128 dev eth0 [centos@test ~]$ sudo rm -rf /var/lib/dhclient/dhclient6--eth0.lease [centos@test ~]$ sudo /sbin/dhclient -6 -1 -lf /var/lib/dhclient/dhclient6--eth0.lease -pf /var/run/dhclient6-eth0.pid eth0 -H test [centos@test ~]$ ip a s eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 - link/ether fa:16:3e:2e:66:ac brd ff:ff:ff:ff:ff:ff - inet 10.136.17.163/22 brd 10.136.19.255 scope global dynamic eth0 - valid_lft 86319sec preferred_lft 86319sec - inet6 2001:db8:123::222/128 scope global dynamic - valid_lft 7481sec preferred_lft 7181sec - inet6 fe80::f816:3eff:fe2e:66ac/64 scope link - valid_lft forever preferred_lft forever + link/ether fa:16:3e:2e:66:ac brd ff:ff:ff:ff:ff:ff + inet 10.136.17.163/22 brd 10.136.19.255 scope global dynamic eth0 + valid_lft 86319sec preferred_lft 86319sec + inet6 2001:db8:123::222/128 scope global dynamic + valid_lft 7481sec preferred_lft 7181sec + inet6 fe80::f816:3eff:fe2e:66ac/64 scope link + valid_lft forever preferred_lft forever [centos@test ~]$ - I found some logic with dhcpv6 leases removing here: - https://opendev.org/openstack/neutron/src/branch/master/neutron/agent/linux/dhcp.py#L600 - but it looks like it doesn't help in case of DHCPCONFIRM client request: + https://opendev.org/openstack/neutron/src/commit/e7b70521d0e230143a80974e7e4795a2acafcc9b/neutron/agent/linux/dhcp.py#L600 + but it looks like it doesn't help in case of DHCPCONFIRM client request: In the dnsmasq logs I see the following DHCPCONFIRM->DHCPREPLY messages exchange after the VM came back after the reboot (see also https://datatracker.ietf.org/doc/html/rfc3315#page-50): Feb 1 16:49:12 dnsmasq-dhcp[1360521]: DHCPREQUEST(tapc233cb5c-8f) 10.136.17.163 fa:16:3e:2e:66:ac Feb 1 16:49:12 dnsmasq-dhcp[1360521]: DHCPACK(tapc233cb5c-8f) 10.136.17.163 fa:16:3e:2e:66:ac host-10-136-17-163 Feb 1 16:49:15 dnsmasq-dhcp[1360521]: DHCPCONFIRM(tapc233cb5c-8f) 00:01:00:01:29:8c:20:5e:fa:16:3e:2e:66:ac Feb 1 16:49:15 dnsmasq-dhcp[1360521]: DHCPREPLY(tapc233cb5c-8f) 2001:db8:123::111 00:01:00:01:29:8c:20:5e:fa:16:3e:2e:66:ac host-2001-db8-123--222 -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1959697 Title: VM gets wrong ipv6 address from dhcp-agent after ipv6 address on port was changed Status in neutron: New Bug description: I run into a problem when neutron dhcp-agent is still replying to the old address confirmation. Simple steps to reproduce: - create a port with IPv6 address in dhcpv6-stateful subnet - create a VM with cloud-init inside - change the IPv6 port address - reboot the VM Here are my commands: $ openstack subnet create --subnet-range 2001:db8:123::/64 --ip-version 6 --ipv6-address-mode dhcpv6-stateful --network public subv6 $ openstack subnet list --network public +--------------------------------------+-------+--------------------------------------+-------------------+ | ID | Name | Network | Subnet | +--------------------------------------+-------+--------------------------------------+-------------------+ | 6d9a7fb5-5c1b-4759-b32b-5720b5cedbf4 | subv4 | f1f3d967-26db-41b3-b6f6-1d5356e33a84 | 10.136.16.0/22 | | 76db898c-6a7a-4301-9253-23241cafaa83 | subv6 | f1f3d967-26db-41b3-b6f6-1d5356e33a84 | 2001:db8:123::/64 | +--------------------------------------+-------+--------------------------------------+-------------------+ $ $ openstack port create my-port --network public --fixed-ip ip-address=10.136.17.163 --fixed-ip ip-address=2001:db8:123::111 $ openstack server create test --flavor m1.small --port my-port --image CentOS-7-x86_64-GenericCloud-2009.qcow2 --key-name key --use-config-drive Check IPv6 address inside VM (it's correct): [centos@test ~]$ ip a s eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether fa:16:3e:2e:66:ac brd ff:ff:ff:ff:ff:ff inet 10.136.17.163/22 brd 10.136.19.255 scope global dynamic eth0 valid_lft 86371sec preferred_lft 86371sec inet6 2001:db8:123::111/128 scope global dynamic valid_lft 7473sec preferred_lft 7173sec inet6 fe80::f816:3eff:fe2e:66ac/64 scope link valid_lft forever preferred_lft forever [centos@test ~]$ Change IPv6 address and reboot the VM: $ openstack port set my-port --no-fixed-ip --fixed-ip ip-address=10.136.17.163 --fixed-ip ip-address=2001:db8:123::222 $ openstack server reboot test [centos@test ~]$ ip a s eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether fa:16:3e:2e:66:ac brd ff:ff:ff:ff:ff:ff inet 10.136.17.163/22 brd 10.136.19.255 scope global dynamic eth0 valid_lft 86382sec preferred_lft 86382sec inet6 2001:db8:123::111/128 scope global dynamic valid_lft 7482sec preferred_lft 7182sec inet6 fe80::f816:3eff:fe2e:66ac/64 scope link valid_lft forever preferred_lft forever [centos@test ~]$ ^^ you can see the VM got the old IPv6 address and actually all traffic is blocked by port-security feature. If I remove a lease file and re-spawn a dhclient, all is fine: [centos@test ~]$ ps axf | grep dhcl 780 ? Ss 0:00 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient--eth0.lease -pf /var/run/dhclient-eth0.pid -H test eth0 868 ? Ss 0:00 /sbin/dhclient -6 -1 -lf /var/lib/dhclient/dhclient6--eth0.lease -pf /var/run/dhclient6-eth0.pid eth0 -H test 1371 pts/0 S+ 0:00 \_ grep --color=auto dhcl [centos@test ~]$ sudo kill -9 868 [centos@test ~]$ sudo ip addr del 2001:db8:123::111/128 dev eth0 [centos@test ~]$ sudo rm -rf /var/lib/dhclient/dhclient6--eth0.lease [centos@test ~]$ sudo /sbin/dhclient -6 -1 -lf /var/lib/dhclient/dhclient6--eth0.lease -pf /var/run/dhclient6-eth0.pid eth0 -H test [centos@test ~]$ ip a s eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether fa:16:3e:2e:66:ac brd ff:ff:ff:ff:ff:ff inet 10.136.17.163/22 brd 10.136.19.255 scope global dynamic eth0 valid_lft 86319sec preferred_lft 86319sec inet6 2001:db8:123::222/128 scope global dynamic valid_lft 7481sec preferred_lft 7181sec inet6 fe80::f816:3eff:fe2e:66ac/64 scope link valid_lft forever preferred_lft forever [centos@test ~]$ I found some logic with dhcpv6 leases removing here: https://opendev.org/openstack/neutron/src/commit/e7b70521d0e230143a80974e7e4795a2acafcc9b/neutron/agent/linux/dhcp.py#L600 but it looks like it doesn't help in case of DHCPCONFIRM client request: In the dnsmasq logs I see the following DHCPCONFIRM->DHCPREPLY messages exchange after the VM came back after the reboot (see also https://datatracker.ietf.org/doc/html/rfc3315#page-50): Feb 1 16:49:12 dnsmasq-dhcp[1360521]: DHCPREQUEST(tapc233cb5c-8f) 10.136.17.163 fa:16:3e:2e:66:ac Feb 1 16:49:12 dnsmasq-dhcp[1360521]: DHCPACK(tapc233cb5c-8f) 10.136.17.163 fa:16:3e:2e:66:ac host-10-136-17-163 Feb 1 16:49:15 dnsmasq-dhcp[1360521]: DHCPCONFIRM(tapc233cb5c-8f) 00:01:00:01:29:8c:20:5e:fa:16:3e:2e:66:ac Feb 1 16:49:15 dnsmasq-dhcp[1360521]: DHCPREPLY(tapc233cb5c-8f) 2001:db8:123::111 00:01:00:01:29:8c:20:5e:fa:16:3e:2e:66:ac host-2001-db8-123--222 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1959697/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp