Reviewed: https://review.openstack.org/268348 Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=303cbc6b5b745aadf026627c4cf2e90e3862b586 Submitter: Jenkins Branch: master
commit 303cbc6b5b745aadf026627c4cf2e90e3862b586 Author: Assaf Muller <[email protected]> Date: Fri Jan 15 16:47:39 2016 -0500 Fix L3 HA with IPv6 We currently use garp_master_repeat and garp_master_refresh to solve bug 1453855. We need to spawn keepalived only after all of the qr/qg ports have been wired so that the initial GARP will be properly sent. Otherwise you get a routing black hole. In lieu of a proper sync method, we used those two keepalived options to send GARPs repeatedly: a) We did not know it never stops spamming the network b) It causes VMs to lose their IPv6 default gateway due to a keepalived bug, which has since been fixed, but it would need to be backported to every keepalived version on every distro. Here's the patch: https://github.com/acassen/keepalived/pull/200 The solution this patch proposes is to drop the repeat and refresh keepalived options. This will fix the IPv6 bug but re-introduce bug 1453855. So, this patch uses the delay option instead. It turns out keepalived sends a GARP when it transitions to MASTER, and then it waits a number of seconds determined by the delay option, and sends a GARP again. We'll use an aggressive 'delay' setting to make sure that when the node boots and the L3/L2 agents start, we'll give the L2 agent enough time to wire the ports as a stopgap solution. Note that this only affects initial synchronization time, not failover times. Failover times will continue to be fast because the ports are wired ahead of time, the initial GARP after the state transition to MASTER will be sent properly. Change-Id: I7a086472b8742828dae08ffd915c45e94fb4b94e Closes-Bug: #1520517 Related-Bug: #1453855 ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1520517 Title: IPv6 networking broken by garp_master_* keepalived settings when using HA routers Status in neutron: Fix Released Bug description: http://git.openstack.org/cgit/openstack/neutron/commit/?id=5d38dc5 added the "garp_master_repeat 5" and "garp_master_refresh 10". This badly breaks networking to the point where it is completely unsuable: First of all, this setting causes Keepalived to constantly spam five unsolicited neighbour advertisements every ten seconds, as shown in this tcpdump (the active router is fe80::f816:3eff:feb8:3857): ubuntu@test:~$ sudo tcpdump -i eth0 host fe80::f816:3eff:feb8:3857 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes 10:08:09.459090 IP6 fe80::f816:3eff:feb8:3857 > ip6-allnodes: ICMP6, router advertisement, length 56 10:08:12.566305 IP6 fe80::f816:3eff:feb8:3857 > ip6-allnodes: ICMP6, router advertisement, length 56 10:08:15.638044 IP6 fe80::f816:3eff:feb8:3857 > ip6-allnodes: ICMP6, router advertisement, length 56 10:08:18.039185 IP6 fe80::f816:3eff:feb8:3857 > ip6-allnodes: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:feb8:3857, length 32 10:08:18.039275 IP6 fe80::f816:3eff:feb8:3857 > ip6-allnodes: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:feb8:3857, length 32 10:08:18.039496 IP6 fe80::f816:3eff:feb8:3857 > ip6-allnodes: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:feb8:3857, length 32 10:08:18.039581 IP6 fe80::f816:3eff:feb8:3857 > ip6-allnodes: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:feb8:3857, length 32 10:08:18.039595 IP6 fe80::f816:3eff:feb8:3857 > ip6-allnodes: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:feb8:3857, length 32 10:08:21.952451 IP6 fe80::f816:3eff:feb8:3857 > ip6-allnodes: ICMP6, router advertisement, length 56 10:08:28.046863 IP6 fe80::f816:3eff:feb8:3857 > ip6-allnodes: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:feb8:3857, length 32 10:08:28.046944 IP6 fe80::f816:3eff:feb8:3857 > ip6-allnodes: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:feb8:3857, length 32 10:08:28.047045 IP6 fe80::f816:3eff:feb8:3857 > ip6-allnodes: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:feb8:3857, length 32 10:08:28.047986 IP6 fe80::f816:3eff:feb8:3857 > ip6-allnodes: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:feb8:3857, length 32 10:08:28.048033 IP6 fe80::f816:3eff:feb8:3857 > ip6-allnodes: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:feb8:3857, length 32 10:08:30.931114 IP6 fe80::f816:3eff:feb8:3857 > ip6-allnodes: ICMP6, router advertisement, length 56 That's not a problem in itself. The problem is however that these neighbour advertisements causes the instance to loose its default route, which stays gone until the next router advertisement packet arrives: ubuntu@test:~$ sudo ip -6 monitor route | ts Nov 27 10:08:09 default via fe80::f816:3eff:feb8:3857 dev eth0 proto ra metric 1024 Nov 27 10:08:18 Deleted default via fe80::f816:3eff:feb8:3857 dev eth0 proto ra metric 1024 expires 27sec hoplimit 64 Nov 27 10:08:21 default via fe80::f816:3eff:feb8:3857 dev eth0 proto ra metric 1024 Nov 27 10:08:28 Deleted default via fe80::f816:3eff:feb8:3857 dev eth0 proto ra metric 1024 expires 23sec hoplimit 64 Nov 27 10:08:30 default via fe80::f816:3eff:feb8:3857 dev eth0 proto ra metric 1024 These periods without a default route obviously badly breaks network connectivity for the node, which is easily observable by a simple ping test: ubuntu@test:~$ ping6 2a02:c0::1 2>&1 | ts Nov 27 10:08:15 PING 2a02:c0::1(2a02:c0::1) 56 data bytes Nov 27 10:08:15 64 bytes from 2a02:c0::1: icmp_seq=1 ttl=62 time=0.570 ms Nov 27 10:08:16 64 bytes from 2a02:c0::1: icmp_seq=2 ttl=62 time=0.873 ms Nov 27 10:08:17 64 bytes from 2a02:c0::1: icmp_seq=3 ttl=62 time=0.666 ms Nov 27 10:08:18 ping: sendmsg: Network is unreachable Nov 27 10:08:19 ping: sendmsg: Network is unreachable Nov 27 10:08:20 ping: sendmsg: Network is unreachable Nov 27 10:08:21 ping: sendmsg: Network is unreachable Nov 27 10:08:22 64 bytes from 2a02:c0::1: icmp_seq=8 ttl=62 time=1.42 ms Nov 27 10:08:23 64 bytes from 2a02:c0::1: icmp_seq=9 ttl=62 time=0.785 ms Nov 27 10:08:24 64 bytes from 2a02:c0::1: icmp_seq=10 ttl=62 time=0.712 ms Nov 27 10:08:25 64 bytes from 2a02:c0::1: icmp_seq=11 ttl=62 time=0.724 ms Nov 27 10:08:26 64 bytes from 2a02:c0::1: icmp_seq=12 ttl=62 time=0.864 ms Nov 27 10:08:27 64 bytes from 2a02:c0::1: icmp_seq=13 ttl=62 time=0.652 ms Nov 27 10:08:28 ping: sendmsg: Network is unreachable Nov 27 10:08:29 ping: sendmsg: Network is unreachable Nov 27 10:08:30 ping: sendmsg: Network is unreachable Nov 27 10:08:31 64 bytes from 2a02:c0::1: icmp_seq=17 ttl=62 time=1.50 ms Nov 27 10:08:32 64 bytes from 2a02:c0::1: icmp_seq=18 ttl=62 time=0.683 ms Nov 27 10:08:33 64 bytes from 2a02:c0::1: icmp_seq=19 ttl=62 time=0.677 ms Nov 27 10:08:34 64 bytes from 2a02:c0::1: icmp_seq=20 ttl=62 time=0.729 ms Removing the garp_master_* settings from keepalived.conf and HUP-ing it solves the problem and makes the network usable again. So the question you might be asking yourself at this point is probably "why do the neighbour advertisement packets cause the default route to be removed"? I am 99.9% certain it is because the NAs have the "router" flag set to 0. (I have not verified it 100%, because I do not know how to configure Keepalived to set the "router" flag, if that is even possible.) ubuntu@test:~$ sudo tshark -i eth0 -f 'icmp6 and ip6[40] == 136 and src host fe80::f816:3eff:feb8:3857' -c 1 -V tshark: Lua: Error during loading: [string "/usr/share/wireshark/init.lua"]:46: dofile has been disabled due to running Wireshark as superuser. See http://wiki.wireshark.org/CaptureSetup/CapturePrivileges for help in running Wireshark as an unprivileged user. Running as user "root" and group "root". This could be dangerous. Capturing on 'eth0' Frame 1: 86 bytes on wire (688 bits), 86 bytes captured (688 bits) on interface 0 Interface id: 0 Encapsulation type: Ethernet (1) Arrival Time: Nov 27, 2015 10:18:48.182563000 UTC [Time shift for this packet: 0.000000000 seconds] Epoch Time: 1448619528.182563000 seconds [Time delta from previous captured frame: 0.000000000 seconds] [Time delta from previous displayed frame: 0.000000000 seconds] [Time since reference or first frame: 0.000000000 seconds] Frame Number: 1 Frame Length: 86 bytes (688 bits) Capture Length: 86 bytes (688 bits) [Frame is marked: False] [Frame is ignored: False] [Protocols in frame: eth:ipv6:icmpv6] Ethernet II, Src: fa:16:3e:b8:38:57 (fa:16:3e:b8:38:57), Dst: IPv6mcast_00:00:00:01 (33:33:00:00:00:01) Destination: IPv6mcast_00:00:00:01 (33:33:00:00:00:01) Address: IPv6mcast_00:00:00:01 (33:33:00:00:00:01) .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default) .... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast) Source: fa:16:3e:b8:38:57 (fa:16:3e:b8:38:57) Address: fa:16:3e:b8:38:57 (fa:16:3e:b8:38:57) .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) Type: IPv6 (0x86dd) Internet Protocol Version 6, Src: fe80::f816:3eff:feb8:3857 (fe80::f816:3eff:feb8:3857), Dst: ff02::1 (ff02::1) 0110 .... = Version: 6 [0110 .... = This field makes the filter "ip.version == 6" possible: 6] .... 0000 0000 .... .... .... .... .... = Traffic class: 0x00000000 .... 0000 00.. .... .... .... .... .... = Differentiated Services Field: Default (0x00000000) .... .... ..0. .... .... .... .... .... = ECN-Capable Transport (ECT): Not set .... .... ...0 .... .... .... .... .... = ECN-CE: Not set .... .... .... 0000 0000 0000 0000 0000 = Flowlabel: 0x00000000 Payload length: 32 Next header: ICMPv6 (58) Hop limit: 255 Source: fe80::f816:3eff:feb8:3857 (fe80::f816:3eff:feb8:3857) Destination: ff02::1 (ff02::1) [Source GeoIP: Unknown] [Destination GeoIP: Unknown] Internet Control Message Protocol v6 Type: Neighbor Advertisement (136) Code: 0 Checksum: 0x0c2b [correct] Flags: 0x20000000 0... .... .... .... .... .... .... .... = Router: Not set <--------------- HERE .0.. .... .... .... .... .... .... .... = Solicited: Not set ..1. .... .... .... .... .... .... .... = Override: Set ...0 0000 0000 0000 0000 0000 0000 0000 = Reserved: 0 Target Address: fe80::f816:3eff:feb8:3857 (fe80::f816:3eff:feb8:3857) ICMPv6 Option (Target link-layer address : fa:16:3e:b8:38:57) Type: Target link-layer address (2) Length: 1 (8 bytes) Link-layer address: fa:16:3e:b8:38:57 (fa:16:3e:b8:38:57) Tore To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1520517/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
