Reviewed: https://review.opendev.org/c/openstack/neutron/+/820897 Committed: https://opendev.org/openstack/neutron/commit/7aae31c9f9ed938760ca0be3c461826b598c7004 Submitter: "Zuul (22348)" Branch: master
commit 7aae31c9f9ed938760ca0be3c461826b598c7004 Author: Bence Romsics <bence.roms...@gmail.com> Date: Tue Oct 5 17:02:41 2021 +0200 Make the dead vlan actually dead All ports plugged into the dead vlan (DEAD_VLAN_TAG 4095 or 0xfff) should not be able to send or receive traffic. We install a flow to br-int to drop all traffic of the dead vlan [1]. However before this patch the flow we install looks like: priority=65535,vlan_tci=0x0fff/0x1fff actions=drop Which is wrong and it usually does not match anything. According to ovs-fields (7) section Open vSwitch Extension VLAN Field, VLAN TCI Field [2] (see especially the usage example vlan_tci=0x1123/0x1fff) we need to explicitly set the bit 0x1000 to match the presence of an 802.1Q header. Setting that bit this flow becomes: priority=65535,vlan_tci=0x1fff/0x1fff actions=drop which is equivalent to: priority=65535,dl_vlan=4095 actions=drop which should match and drop dead vlan traffic. However there's a second problem: ovs access ports were designed to work together with the NORMAL action. The NORMAL action considers the vlan of an access port, but the openflow pipeline does not. An openflow rule does not see the vlan set for an access port, because that vlan tag is only pushed to the frame if and when the frame leaves the switch on a trunk port [3][4]. So we have to explicitly push the DEAD_VLAN_TAG if we want the dead vlan's drop flow match anything. That means we are adding a flow to push the dead vlan tag from dhcp-agent/l3-agent but we are deleting that flow from ovs-agent right after ovs-agent sets the vlan tag of the port to a non-dead vlan. Which is ugly but we have to keep adding the flow as early as possible if we want to minimize the window until frames can leak onto the dead vlan. Even with this change there's a short time window in which the dead vlan could theoretically leak. [1] https://opendev.org/openstack/neutron/src/commit/ecdc11a56448428f77f5a64fd028f1e4c9644ea3/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/br_int.py#L60-L62 [2] http://www.openvswitch.org/support/dist-docs/ovs-fields.7.html [3] https://mail.openvswitch.org/pipermail/ovs-discuss/2021-December/051647.html [4] https://docs.openvswitch.org/en/latest/faq/vlan/ see 'Q: My OpenFlow controller doesn’t see the VLANs that I expect.' Change-Id: Ib6b70114efb140cf1393b57ebc350fea4b0a2443 Closes-Bug: #1930414 ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1930414 Title: Traffic leaked from dhcp port before vlan tag is applied Status in neutron: Fix Released Status in OpenStack Security Advisory: Won't Fix Bug description: This is a bug with potential security implications. I don't see a clear way to exploit it at the moment, but to err on the safe side, I'm opening this as private to the security team. Short summary: Using openvswitch-agent, traffic sent on some (at least dhcp) ports before ovs-agent applies the port's vlan tag can be seen and intercepted on ports from other networks on the same integration bridge. We observed this bug: * using vlan and vxlan networks * using the noop and openvswitch firewall drivers * on openstack versions mitaka, pike and master (commit 5a6f61af4a) The time window between the port's creation and ovs-agent applying its vlan tag is usually very short. We observed this bug in the wild on a heavily loaded host. However to make the reproduction reliable on lightly loaded systems I inserted a sleep() into ovs-agent's source (just before the port's vlan tag is set): $ git --no-pager format-patch --stdout 5a6f61af4a From 8389b3e8e5c60c81ff2bb262e3ae2e8aab73d3f5 Mon Sep 17 00:00:00 2001 From: Bence Romsics <bence.roms...@gmail.com> Date: Mon, 31 May 2021 13:12:34 +0200 Subject: [PATCH] WIP Change-Id: Ibef4278a2f6a85f52a8ffa43caef6de38cbb40cb --- .../plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py | 1 + 1 file changed, 1 insertion(+) diff --git a/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py b/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py index 2c209bd387..355584b325 100644 --- a/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py +++ b/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py @@ -1190,6 +1190,7 @@ class OVSNeutronAgent(l2population_rpc.L2populationRpcCallBackTunnelMixin, self.setup_arp_spoofing_protection(self.int_br, port, port_detail) if cur_tag != lvm.vlan: + time.sleep(3000) self.int_br.set_db_attribute( "Port", port.port_name, "tag", lvm.vlan) · --· 2.31.0 We discovered the bug by the following procedure: * a test environment created multiple neutron nets in short succession with the exact same ipv6 subnet ranges * therefore neutron selected the exact same ipv6 address for the subnet's dhcp port * the host running the dhcp-agent and the ovs-agent was heavily loaded * we observed that many (when ovs-agent is made slow enough, then all but one) of these networks' services relying on the dhcp port's address were unavailable * because duplicate address detection (DAD) for the ipv6 dhcp port address failed * we believe DAD failed because we have some temporary crosstalk between the dhcp namespaces of different networks * we believe that this bug is not ipv6 specific, only the default DAD of ipv6 helped us discover it Exact reproduction steps: $ date --iso-8601=s 2021-05-31T13:10:14+00:00 # when the ovs-agent is slow enough, even 2 networks are sufficient $ for i in {1..5} do openstack network create xnet$i openstack subnet create xsubnet$i-v6 --ip-version 6 --network xnet$i --subnet-range 2001:db8::/32 done # for the record $ openstack subnet list -f value -c Name -c ID | egrep xsubnet 01d614da-820b-418d-8fa4-71952713f0ad xsubnet5-v6 72158e8e-5059-4abb-98a4-5adc9e4ef39c xsubnet2-v6 8f263143-a69b-4c42-b74c-6f30aca7b19d xsubnet4-v6 9ab4159e-12f8-44ed-8947-35a56b62eaf8 xsubnet1-v6 d4ed53e2-7b70-43d7-bd9f-d45f006a8179 xsubnet3-v6 # note that all dhcp ports got the same ip $ openstack port list --device-owner network:dhcp -f value -c id -c mac_address -c fixed_ips | egrep 2001:db8:: 130855be-ead1-40bb-9ca0-5336428aa74b fa:16:3e:24:76:41 [{'subnet_id': '01d614da-820b-418d-8fa4-71952713f0ad', 'ip_address': '2001:db8::1'}] 19fcabfd-f32a-40ea-b68e-ced41f394822 fa:16:3e:43:80:fe [{'subnet_id': '9ab4159e-12f8-44ed-8947-35a56b62eaf8', 'ip_address': '2001:db8::1'}] 46963dbd-c844-4986-a07f-fb78adbd95e9 fa:16:3e:4f:23:bf [{'subnet_id': '72158e8e-5059-4abb-98a4-5adc9e4ef39c', 'ip_address': '2001:db8::1'}] b8bf2fb1-d52a-41af-90bb-01aa23529015 fa:16:3e:90:40:8e [{'subnet_id': 'd4ed53e2-7b70-43d7-bd9f-d45f006a8179', 'ip_address': '2001:db8::1'}] ba67f2c0-c714-45fd-aec8-7233ba379dfa fa:16:3e:35:10:8d [{'subnet_id': '8f263143-a69b-4c42-b74c-6f30aca7b19d', 'ip_address': '2001:db8::1'}] # all but one dhcp port (and by the way metadata) addresses are in DAD failed mode $ for net in $( openstack network list -f value -c Name -c ID | awk '/ xnet/ { print $1 }' ) ; do sudo ip netns exec qdhcp-$net ip a ; done | egrep '(link/ether|inet6 (2001:db8::|fe80::a9fe:a9fe))' link/ether fa:16:3e:90:40:8e brd ff:ff:ff:ff:ff:ff inet6 2001:db8::1/32 scope global dadfailed tentative· inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative· link/ether fa:16:3e:4f:23:bf brd ff:ff:ff:ff:ff:ff inet6 2001:db8::1/32 scope global dadfailed tentative· inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative· link/ether fa:16:3e:24:76:41 brd ff:ff:ff:ff:ff:ff inet6 2001:db8::1/32 scope global dadfailed tentative· inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative· link/ether fa:16:3e:35:10:8d brd ff:ff:ff:ff:ff:ff inet6 2001:db8::1/32 scope global dadfailed tentative· inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative· link/ether fa:16:3e:43:80:fe brd ff:ff:ff:ff:ff:ff inet6 2001:db8::1/32 scope global· inet6 fe80::a9fe:a9fe/64 scope link· # dmesg also shows the DAD failures # man dmesg: Be aware that the timestamp could be inaccurate! $ LC_TIME=en_US sudo dmesg -T [snip] [Mon May 31 13:10:10 2021] device tap19fcabfd-f3 entered promiscuous mode [Mon May 31 13:10:15 2021] device tap46963dbd-c8 entered promiscuous mode [Mon May 31 13:10:15 2021] IPv6: tap46963dbd-c8: IPv6 duplicate address fe80::a9fe:a9fe used by fa:16:3e:43:80:fe detected! [Mon May 31 13:10:15 2021] IPv6: tap46963dbd-c8: IPv6 duplicate address 2001:db8::1 used by fa:16:3e:43:80:fe detected! [Mon May 31 13:10:18 2021] device tapb8bf2fb1-d5 entered promiscuous mode [Mon May 31 13:10:20 2021] IPv6: tapb8bf2fb1-d5: IPv6 duplicate address 2001:db8::1 used by fa:16:3e:43:80:fe detected! [Mon May 31 13:10:20 2021] IPv6: tapb8bf2fb1-d5: IPv6 duplicate address fe80::a9fe:a9fe used by fa:16:3e:43:80:fe detected! [Mon May 31 13:10:22 2021] device tapba67f2c0-c7 entered promiscuous mode [Mon May 31 13:10:23 2021] IPv6: tapba67f2c0-c7: IPv6 duplicate address 2001:db8::1 used by fa:16:3e:43:80:fe detected! [Mon May 31 13:10:24 2021] IPv6: tapba67f2c0-c7: IPv6 duplicate address fe80::a9fe:a9fe used by fa:16:3e:43:80:fe detected! [Mon May 31 13:10:26 2021] device tap130855be-ea entered promiscuous mode [Mon May 31 13:10:27 2021] IPv6: tap130855be-ea: IPv6 duplicate address 2001:db8::1 used by fa:16:3e:43:80:fe detected! [Mon May 31 13:10:28 2021] IPv6: tap130855be-ea: IPv6 duplicate address fe80::a9fe:a9fe used by fa:16:3e:43:80:fe detected! # while there were no errors in ovs-agent $ sudo LC_TIME=en_US journalctl -u devstack@q-agt -S '2021-05-31 13:10:14' | egrep ERROR [empty] # clean up $ openstack network list | awk '/xnet/ { print $2 }' | xargs -r openstack network delete In a bit of further analysis I only created two networks and ran tcpdump on the first's dhcp port, while creating the second: $ openstack network create xnet0 $ openstack subnet create xsubnet0-v6 --ip-version 6 --network xnet0 --subnet-range 2001:db8::/32 # run tcpdump on the 1st net's dhcp port $ sudo ip netns exec qdhcp-$( openstack network show -f value -c id xnet0 ) tcpdump -n -vvv -i tapcaa92c34-53 # create the 2nd net while tcpdump is already running $ openstack network create xnet1 $ openstack subnet create xsubnet1-v6 --ip-version 6 --network xnet1 --subnet-range 2001:db8::/32 # the 2nd net's dhcp port's mac address $ openstack port list --device-owner network:dhcp --network xnet1 -f value -c mac_address fa:16:3e:0d:be:aa # tcpdump's capture contains packets with the 2nd net's dhcp port's mac address tcpdump: listening on tapcaa92c34-53, link-type EN10MB (Ethernet), capture size 262144 bytes ^C14:24:14.541893 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 90: (hlim 1, next-header Options (0) payload length: 36) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 1 group record(s) [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)] 14:24:14.929686 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 110: (hlim 1, next-header Options (0) payload length: 56) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 2 group record(s) [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)] 14:24:14.985673 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 130: (hlim 1, next-header Options (0) payload length: 76) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 3 group record(s) [gaddr ff02::1:ff00:1 to_ex, 0 source(s)] [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)] 14:24:14.993930 fa:16:3e:0d:be:aa > 33:33:ff:0d:be:aa, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) :: > ff02::1:ff0d:beaa: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::f816:3eff:fe0d:beaa unknown option (14), length 8 (1):· 0x0000: bfdb db94 d695 14:24:15.209668 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 130: (hlim 1, next-header Options (0) payload length: 76) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 3 group record(s) [gaddr ff02::1:ff00:1 to_ex, 0 source(s)] [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)] 14:24:15.858999 fa:16:3e:0d:be:aa > 33:33:ff:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) :: > ff02::1:ff00:1: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:db8::1 unknown option (14), length 8 (1):· 0x0000: 6e67 6a99 e72e 14:24:15.859815 fa:16:3e:6a:ed:cb > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) 2001:db8::1 > ff02::1: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is 2001:db8::1, Flags [override] destination link-address option (2), length 8 (1): fa:16:3e:6a:ed:cb 14:24:15.860188 fa:16:3e:0d:be:aa > 33:33:ff:fe:a9:fe, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) :: > ff02::1:fffe:a9fe: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::a9fe:a9fe unknown option (14), length 8 (1):· 0x0000: 11e1 9aab 157a 14:24:15.860731 fa:16:3e:6a:ed:cb > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::a9fe:a9fe > ff02::1: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is fe80::a9fe:a9fe, Flags [override] destination link-address option (2), length 8 (1): fa:16:3e:6a:ed:cb 14:24:16.017855 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 130: (hlim 1, next-header Options (0) payload length: 76) fe80::f816:3eff:fe0d:beaa > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 3 group record(s) [gaddr ff02::1:ff00:1 to_ex, 0 source(s)] [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)] 14:24:16.137879 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 130: (hlim 1, next-header Options (0) payload length: 76) fe80::f816:3eff:fe0d:beaa > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 3 group record(s) [gaddr ff02::1:ff00:1 to_ex, 0 source(s)] [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)] 11 packets captured 11 packets received by filter 0 packets dropped by kernel At the moment I did not test yet: * whether the leaked traffic can be intercepted on a vm port * whether a vm port can similarly leak traffic * whether ovs-agent can be attacked to intentionally slow it down To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1930414/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp