Public bug reported: * High level description: linux bridge assigns mac address to the physical external interface instead of the tap interface, therefore the VM instance that uses the tap interface is not able to communicate over IP. The workaround I have found is to convert the bridge to a hub, by setting ageing to 0 (brctl setageing br-name 0). In this way, the bridge floods all packets on all attached bridge interfaces, and everything starts working.
* Pre-conditions: I have an openstack pike running in latest centos 7 release (7.4.1708). Neutron was manually installed as described in the neutron installation guide at https://docs.openstack.org/neutron/latest/install/install-rdo.html. I have configured neutron for Network Option 2 (self service networks), however the setup I am testing here is an external flat provider network with a single cirros VM instance attached directly to it (without any router in between). The openstack environment is made of two nodes: a controller and a compute. The neutron package versions is 11.0.2-2.el7 (latest in centos 7), the bridge-utils version is 1.5-9.el7 and the kernel version is 3.10.0-693.11.1.el7.x86_64. I have tested this with cirros image cirros-0.4.0-x86_64-disk.img and cirros-0.3.5-x86_64-disk.img. # rpm -qa | grep neutron-linuxbridge openstack-neutron-linuxbridge-11.0.2-2.el7.noarch # rpm -qf /usr/sbin/brctl bridge-utils-1.5-9.el7.x86_64 # uname -a Linux compute1 3.10.0-693.11.1.el7.x86_64 #1 SMP Mon Dec 4 23:52:40 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Bridging is configured like below in both controller and compute: ml2_conf.ini: [ml2_type_flat] flat_networks = physnet1 linuxbridge_agent.ini: [linux_bridge] physical_interface_mappings = physnet1:bond2 * Step-by-step reproduction steps: This is how I created the provider network: openstack network create \ --share \ --external \ --provider-physical-network physnet1 \ --provider-network-type flat \ ExtNet1 This is how I create the provider subnet: openstack subnet create \ --network ExtNet1 \ --allocation-pool start=10.20.21.96,end=$10.20.21.127 \ --dns-nameserver 10.20.21.1 \ --gateway 10.20.21.1 \ --subnet-range 10.20.21.0/24 \ ExtSubnet1 This is how I launch a cirros instance and attach it to the provider network: openstack server create \ --flavor m1.nano \ --image cirros-0.4.0-x86_64-disk.img \ --nic net-id=$(openstack network list | grep ExtNet1 | cut -d\ -f 2) \ --security-group default \ --key-name controller-key \ cirros1 Based on the above, neutron creates in my compute node the following bridge: # brctl show bridge name bridge id STP enabled interfaces brq75a55ef7-4a 8000.fc15b413e6a3 no bond2 tap44bc34bb-e2 bond2 is the physical interface used for the flat provider network (in access mode, no vlans) and tap44bc34bb-e2 is the tap interface attached to my cirros VM instance. In the bridge, the bond2 is port 2, and the tap tap44bc34bb-e2 interface is port 1, and both are in forwarding mode. # brctl showstp brq75a55ef7-4a brq75a55ef7-4a <...> ageing time 300.00 <...> bond2 (2) port id 8002 state forwarding <...> tap44bc34bb-e2 (1) port id 8001 state forwarding <...> The network flow is like below: default gw<-->physical switch<-->[bond2 bridge tap]<-->[eth0 cirrosVM] eth0 mac address is fa:16:3e:cc:dc:ec. After the cirros VM comes up, it is not able to get an IP address from the DHCP agent, and there is no IP communication. Therefore I have to use the console and manually assign the corresponding IP address to the cirros VM eth0 interface, but still the IP connectivity does not work, since I cannot ping any external IPs, even in the same 10.20.21.0/24 subnet. What I have found is that in the bridge forwarding table, the bridge wrongly assigns the eth0 mac address to the port 2, which is bond2 interface, instead of assigning it to the port 1, which is the tap interface. This happens only if the arp table in the cirros VM instance does not contain the mac address of the destination IP I am pinging (default gw in this case), so the cirros VM sends an arp request (Request who-has 10.203.219.1 tell 10.203.219.114). See below the eth0 mac address wrongly assigned in the forwarding table to the port 2: # brctl showmacs brq75a55ef7-4a | grep fa:16:3e:cc:dc:ec 2 fa:16:3e:cc:dc:ec no 0.39 However, since the eth0 mac address is wrongly assigned to the port2, the arp reply back (Reply 10.203.219.1 is-at 00:17:08:c4:52:80) does not reach anymore eth0. Using a tcpdump, I can see in the compute node the arp request going through the tap interface an the bridge, however the arp reply back does not show up anymore on the tap interface. Since the arp reply back does not reach the eth0, the arp table in the cirros VM does not contain the proper entries, and therefore the IP communication (ping in my case) does not work. The strange thing is that after a while, for apparently no reason, a single arp reply back packet gets through the bridge and the tap interface, and the arp table gets updated with correct mac address in the cirros VM instance, and I am able to ping that IP address. If I manually add the mac address of the destination IP I am pining into the cirros VM instance arp table, and there is no arp request sent, just icmp packets going out, then the bridge correctly assigns the eth0 mac address to the port 1, which is the tap interface, and everything starts working fine. See below the eth0 mac address correctly assigned in the forwarding table to the port 1: # brctl showmacs brq75a55ef7-4a | grep fa:16:3e:cc:dc:ec 1 fa:16:3e:cc:dc:ec no 0.09 The only workaround I have found to this issue is to configure the bridge ageing time to zero, converting the bridge into a hub so that it floods all packets on all ports. It is also strange that in the bridge forwarding table, the bond0 and tap interface mac addresses are shown twice, as seen below: # brctl showmacs brq75a55ef7-4a | grep yes 2 fc:15:b4:13:e6:a3 yes 0.00 2 fc:15:b4:13:e6:a3 yes 0.00 1 fe:16:3e:cc:dc:ec yes 0.00 1 fe:16:3e:cc:dc:ec yes 0.00 I have tested this in an all-in-one openstack installation running in a vmware centos 7 VM, that has a single interface configured in promiscuous mode in the vmware portgroup (therefore no interface bonding like in the previous setup), and the same issue happens, therefore I do not think this is caused by the bonding interface. ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1738659 Title: linux bridge assigns mac address to the wrong port Status in neutron: New Bug description: * High level description: linux bridge assigns mac address to the physical external interface instead of the tap interface, therefore the VM instance that uses the tap interface is not able to communicate over IP. The workaround I have found is to convert the bridge to a hub, by setting ageing to 0 (brctl setageing br-name 0). In this way, the bridge floods all packets on all attached bridge interfaces, and everything starts working. * Pre-conditions: I have an openstack pike running in latest centos 7 release (7.4.1708). Neutron was manually installed as described in the neutron installation guide at https://docs.openstack.org/neutron/latest/install/install-rdo.html. I have configured neutron for Network Option 2 (self service networks), however the setup I am testing here is an external flat provider network with a single cirros VM instance attached directly to it (without any router in between). The openstack environment is made of two nodes: a controller and a compute. The neutron package versions is 11.0.2-2.el7 (latest in centos 7), the bridge-utils version is 1.5-9.el7 and the kernel version is 3.10.0-693.11.1.el7.x86_64. I have tested this with cirros image cirros-0.4.0-x86_64-disk.img and cirros-0.3.5-x86_64-disk.img. # rpm -qa | grep neutron-linuxbridge openstack-neutron-linuxbridge-11.0.2-2.el7.noarch # rpm -qf /usr/sbin/brctl bridge-utils-1.5-9.el7.x86_64 # uname -a Linux compute1 3.10.0-693.11.1.el7.x86_64 #1 SMP Mon Dec 4 23:52:40 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Bridging is configured like below in both controller and compute: ml2_conf.ini: [ml2_type_flat] flat_networks = physnet1 linuxbridge_agent.ini: [linux_bridge] physical_interface_mappings = physnet1:bond2 * Step-by-step reproduction steps: This is how I created the provider network: openstack network create \ --share \ --external \ --provider-physical-network physnet1 \ --provider-network-type flat \ ExtNet1 This is how I create the provider subnet: openstack subnet create \ --network ExtNet1 \ --allocation-pool start=10.20.21.96,end=$10.20.21.127 \ --dns-nameserver 10.20.21.1 \ --gateway 10.20.21.1 \ --subnet-range 10.20.21.0/24 \ ExtSubnet1 This is how I launch a cirros instance and attach it to the provider network: openstack server create \ --flavor m1.nano \ --image cirros-0.4.0-x86_64-disk.img \ --nic net-id=$(openstack network list | grep ExtNet1 | cut -d\ -f 2) \ --security-group default \ --key-name controller-key \ cirros1 Based on the above, neutron creates in my compute node the following bridge: # brctl show bridge name bridge id STP enabled interfaces brq75a55ef7-4a 8000.fc15b413e6a3 no bond2 tap44bc34bb-e2 bond2 is the physical interface used for the flat provider network (in access mode, no vlans) and tap44bc34bb-e2 is the tap interface attached to my cirros VM instance. In the bridge, the bond2 is port 2, and the tap tap44bc34bb-e2 interface is port 1, and both are in forwarding mode. # brctl showstp brq75a55ef7-4a brq75a55ef7-4a <...> ageing time 300.00 <...> bond2 (2) port id 8002 state forwarding <...> tap44bc34bb-e2 (1) port id 8001 state forwarding <...> The network flow is like below: default gw<-->physical switch<-->[bond2 bridge tap]<-->[eth0 cirrosVM] eth0 mac address is fa:16:3e:cc:dc:ec. After the cirros VM comes up, it is not able to get an IP address from the DHCP agent, and there is no IP communication. Therefore I have to use the console and manually assign the corresponding IP address to the cirros VM eth0 interface, but still the IP connectivity does not work, since I cannot ping any external IPs, even in the same 10.20.21.0/24 subnet. What I have found is that in the bridge forwarding table, the bridge wrongly assigns the eth0 mac address to the port 2, which is bond2 interface, instead of assigning it to the port 1, which is the tap interface. This happens only if the arp table in the cirros VM instance does not contain the mac address of the destination IP I am pinging (default gw in this case), so the cirros VM sends an arp request (Request who-has 10.203.219.1 tell 10.203.219.114). See below the eth0 mac address wrongly assigned in the forwarding table to the port 2: # brctl showmacs brq75a55ef7-4a | grep fa:16:3e:cc:dc:ec 2 fa:16:3e:cc:dc:ec no 0.39 However, since the eth0 mac address is wrongly assigned to the port2, the arp reply back (Reply 10.203.219.1 is-at 00:17:08:c4:52:80) does not reach anymore eth0. Using a tcpdump, I can see in the compute node the arp request going through the tap interface an the bridge, however the arp reply back does not show up anymore on the tap interface. Since the arp reply back does not reach the eth0, the arp table in the cirros VM does not contain the proper entries, and therefore the IP communication (ping in my case) does not work. The strange thing is that after a while, for apparently no reason, a single arp reply back packet gets through the bridge and the tap interface, and the arp table gets updated with correct mac address in the cirros VM instance, and I am able to ping that IP address. If I manually add the mac address of the destination IP I am pining into the cirros VM instance arp table, and there is no arp request sent, just icmp packets going out, then the bridge correctly assigns the eth0 mac address to the port 1, which is the tap interface, and everything starts working fine. See below the eth0 mac address correctly assigned in the forwarding table to the port 1: # brctl showmacs brq75a55ef7-4a | grep fa:16:3e:cc:dc:ec 1 fa:16:3e:cc:dc:ec no 0.09 The only workaround I have found to this issue is to configure the bridge ageing time to zero, converting the bridge into a hub so that it floods all packets on all ports. It is also strange that in the bridge forwarding table, the bond0 and tap interface mac addresses are shown twice, as seen below: # brctl showmacs brq75a55ef7-4a | grep yes 2 fc:15:b4:13:e6:a3 yes 0.00 2 fc:15:b4:13:e6:a3 yes 0.00 1 fe:16:3e:cc:dc:ec yes 0.00 1 fe:16:3e:cc:dc:ec yes 0.00 I have tested this in an all-in-one openstack installation running in a vmware centos 7 VM, that has a single interface configured in promiscuous mode in the vmware portgroup (therefore no interface bonding like in the previous setup), and the same issue happens, therefore I do not think this is caused by the bonding interface. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1738659/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
