Bug#1071184: Kernel 6.6 and 6.7 route-leak between VRF and default leads to Time to live exceeded
On Friday, 17 May 2024 15:08:17 CEST Development EasyNet wrote: > I will try. Meanwhile I was troubleshooting this issue for some time and > I notice a change in FRRouting between 9.1 and 10.0. > Before 10.0 FRRouting was installing the routes in kernel using the > destination interface of the route. Starting from 10.0 FRRouting is > installing all routes towards the VRF interface. > > Here is my bug reported on FRRouting: > https://github.com/FRRouting/frr/issues/15909 I have no (particular) knowledge about kernel routing or FRRouting, so I can't help with that aspect. But if the problem is resolved with 6.8.9, then that seems the easiest solution and means the underlying issue is fixed. If not, it's useful to know if there is a(n older) kernel version where it does work. But given there's also a FRR 9.x -> 10.x upgrade at play, I'm not so sure the problem is actually in the kernel. signature.asc Description: This is a digitally signed message part.
Bug#1071184: Kernel 6.6 and 6.7 route-leak between VRF and default leads to Time to live exceeded
Hi Diederik, I will try. Meanwhile I was troubleshooting this issue for some time and I notice a change in FRRouting between 9.1 and 10.0. Before 10.0 FRRouting was installing the routes in kernel using the destination interface of the route. Starting from 10.0 FRRouting is installing all routes towards the VRF interface. Here is my bug reported on FRRouting: https://github.com/FRRouting/frr/issues/15909 Example: Working scenario with FRR 9.0.2 and 9.1: |root@FRR01:/opt/Kitts/frr/9.0.2# ip nexthop show id 14 dev lo scope host proto zebra id 15 dev ens33 scope host proto zebra id 16 dev ens36 scope host proto zebra id 17 dev ens37 scope host proto zebra id 18 dev ens38 scope host proto zebra id 19 dev ens33 scope link proto zebra id 21 dev ens36 scope link proto zebra id 23 dev ens37 scope link proto zebra id 25 dev ens38 scope link proto zebra id 26 dev lo3 scope link proto zebra id 30 blackhole proto zebra id 31 blackhole proto zebra id 32 via 192.168.1.1 dev ens33 scope link proto zebra id 36 dev ens37 scope host proto zebra id 37 dev lo scope host proto zebra id 38 dev ens38 scope host proto zebra root@FRR01:/opt/Kitts/frr/9.0.2# ip nexthop show vrf red id 18 dev ens38 scope host proto zebra id 25 dev ens38 scope link proto zebra id 38 dev ens38 scope host proto zebra root@FRR01:/opt/Kitts/frr/9.0.2# ip route list 10.0.0.0/30 dev ens37 proto kernel scope link src 10.0.0.1 10.0.1.0/30 nhid 38 dev ens38 proto bgp metric 20 root@FRR01:/opt/Kitts/frr/9.0.2# ip route show table local local 10.0.0.1 dev ens37 proto kernel scope host src 10.0.0.1 broadcast 10.0.0.3 dev ens37 proto kernel scope link src 10.0.0.1 local 10.100.0.1 dev lo proto kernel scope host src 10.100.0.1 broadcast 10.100.0.1 dev lo proto kernel scope link src 10.100.0.1 local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1 local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1 broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1 root@FRR01:/opt/Kitts/frr/9.0.2# ip route show vrf red blackhole default proto static metric 20 10.0.0.0/30 nhid 36 dev ens37 proto bgp metric 20 10.0.1.0/30 dev ens38 proto kernel scope link src 10.0.1.1 10.100.0.1 nhid 37 dev lo proto bgp metric 20 root@FRR01:/opt/Kitts/frr/9.0.2# ip route show table red blackhole default proto static metric 20 10.0.0.0/30 nhid 36 dev ens37 proto bgp metric 20 10.0.1.0/30 dev ens38 proto kernel scope link src 10.0.1.1 local 10.0.1.1 dev ens38 proto kernel scope host src 10.0.1.1 broadcast 10.0.1.3 dev ens38 proto kernel scope link src 10.0.1.1 10.100.0.1 nhid 37 dev lo proto bgp metric 20 root@FRR01:/opt/Kitts/frr/9.0.2# ip route show vrf red blackhole default proto static metric 20 10.0.0.0/30 nhid 36 dev ens37 proto bgp metric 20 10.0.1.0/30 dev ens38 proto kernel scope link src 10.0.1.1 10.100.0.1 nhid 37 dev lo proto bgp metric 20 root@FRR01:/opt/Kitts/frr/9.0.2# ip rule list 0: from all lookup local 1000: from all lookup [l3mdev-table] 32766: from all lookup main 32767: from all lookup default root@FRR01:/opt/Kitts/frr/9.0.2# Non-working scenario with FRR 10.0: ||root@FRR01:/# ip nexthop show id 2 dev lo0 scope link proto zebra id 4 dev lo1 scope link proto zebra id 6 dev lo2 scope link proto zebra id 8 dev lo3 scope link proto zebra id 10 dev ens36 scope host proto zebra id 17 dev ens37 scope host proto zebra id 18 dev ens38 scope host proto zebra id 19 dev lo scope host proto zebra id 20 dev ens33 scope host proto zebra id 21 blackhole proto zebra id 22 blackhole proto zebra id 24 via 192.168.1.1 dev ens33 scope link proto zebra id 32 dev ens33 scope link proto zebra id 34 dev lo scope host proto zebra id 36 dev red scope host proto zebra root@FRR01:/# ip nexthop show vrf red id 18 dev ens38 scope host proto zebra id 25 dev ens38 scope link proto zebra| |root@FRR01:/# ip route list| |10.0.0.0/30 dev ens37 proto kernel scope link src 10.0.0.1 10.0.1.0/30 nhid 36 dev red proto bgp metric 20 root@FRR01:/# ip route show table local local 10.0.0.1 dev ens37 proto kernel scope host src 10.0.0.1 broadcast 10.0.0.3 dev ens37 proto kernel scope link src 10.0.0.1 local 10.100.0.1 dev lo proto kernel scope host src 10.100.0.1 broadcast 10.100.0.1 dev lo proto kernel scope link src 10.100.0.1 local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1 local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1 broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1 root@FRR01:/# ip route show vrf red blackhole default proto static metric 20 10.0.0.0/30 nhid 34 dev lo proto bgp metric 20 10.0.1.0/30 dev ens38 proto kernel scope link src 10.0.1.1 10.100.0.1 nhid 34 dev lo proto bgp metric 20 root@FRR01:/# ip route show table red blackhole default proto static metric 20 10.0.0.0/30 nhid 34 dev lo proto bgp metric 20 10.0.1.0/30 dev ens38 proto kernel scope link src 10.0.1.1 local 10.0.1.1 dev ens38 proto kernel scope host src 10.0.1.1 broadcast 10.0.1.3 dev ens38
Bug#1071184: Kernel 6.6 and 6.7 route-leak between VRF and default leads to Time to live exceeded
Control: tag -1 moreinfo On 15 May 2024 16:08:27 +0200 Development EasyNet wrote: > Package: linux-image > Version: 6.6.15-2 and 6.7.12-1 > > I'm facing for some time a strange behavior of the route-leak. It happen > on both IPv4 and IPv6. > Configuration used: Debian Trixie, Kernel 6.7.12 with FRRouting 10.1 - git > VRF: internet > Default: just local management Sid recently got a 6.8.9 kernel, can you test whether that fixes the issue? signature.asc Description: This is a digitally signed message part.
Bug#1071184: Kernel 6.6 and 6.7 route-leak between VRF and default leads to Time to live exceeded
Package: linux-image Version: 6.6.15-2 and 6.7.12-1 Hello, I'm facing for some time a strange behavior of the route-leak. It happen on both IPv4 and IPv6. Configuration used: Debian Trixie, Kernel 6.7.12 with FRRouting 10.1 - git VRF: internet Default: just local management Route-Leak between internet <-> default: FRR config: ip route 0.0.0.0/0 internet nexthop-vrf internet .. router bgp 43XXX .. address-family ipv4 unicast redistribute connected route-map VPN-export-GRT-connected no neighbor MPLS-v4 activate no neighbor MPLS-v6 activate no neighbor SPOKES-v4 activate no neighbor SPOKES-v6 activate label vpn export auto rd vpn export 43474:11002 rt vpn import 43XXX:11000 43XXX:11999 rt vpn export 43XXX:11000 export vpn import vpn exit-address-family ! address-family ipv4 vpn neighbor MPLS-v4 activate neighbor MPLS-v4 soft-reconfiguration inbound neighbor SPOKES-v4 activate neighbor SPOKES-v4 soft-reconfiguration inbound exit-address-family router bgp 43XXX vrf internet address-family ipv4 unicast maximum-paths 4 label vpn export auto rd vpn export 43XXX:10002 rt vpn import 43XXX:1 43XXX:10100 43XXX:10200 43XXX:10500 43XXX:10700 43XXX:10800 43XXX:10999 rt vpn export 43XXX:1 export vpn import vpn .. route-map VPN-export-GRT-connected permit 1000 match ip address prefix-list pl-EASYNET-subnets set extcommunity rt 43XXX:10999 exit ! route-map VPN-export-GRT-connected permit 1100 match ipv6 address prefix-list pl-EASYNET-subnets set extcommunity rt 43XXX:10999 exit ! route-map VPN-export-GRT-connected deny 65535 exit ip prefix-list pl-EASYNET-subnets description EASYNET IPv4 subnets ip prefix-list pl-EASYNET-subnets seq 5 permit 89.X.X.0/24 le 32 ipv6 prefix-list pl-EASYNET-subnets description description EASYNET IPv6 subnets ipv6 prefix-list pl-EASYNET-subnets seq 5 permit 2a00::Y::/48 le 128 To be able to export local connected routes from default I'm using the communities to export them into internet VRF. In default I'm using a default route to vrf internet. Route table: S>* 0.0.0.0/0 [1/0] is directly connected, internet (vrf internet), weight 1, 00:06:49 .. R02(config-if)# do sh ip route | include lo Codes: K - kernel route, C - connected, L - local, S - static, t - trapped, o - offload failure O 10.100.2.1/32 [110/0] is directly connected, lo, weight 1, 00:15:41 L * 10.100.2.1/32 is directly connected, lo, 1d14h54m C>* 10.100.2.1/32 is directly connected, lo, 1d14h54m O 89.X.X.2/32 [110/0] is directly connected, lo, weight 1, 00:00:07 L * 89.X.X.2/32 is directly connected, lo, 00:00:07 C>* 89.X.X.2/32 is directly connected, lo, 00:00:07 O 89.X.Y.118/32 [110/0] is directly connected, lo, weight 1, 00:15:41 L * 89.X.Y.118/32 is directly connected, lo, 1d14h54m C>* 89.X.Y.118/32 is directly connected, lo, 1d14h54m R02(config-if)# do sh ip route vrf internet 89.X.X.2/32 Routing entry for 89.X.X.2/32 Known via "bgp", distance 20, metric 0, vrf internet, best Last update 00:00:56 ago * directly connected, lo(vrf default), weight 1 But the TCPDUMP looks like this: root@R02:/home/adrian# tcpdump -nvlei any host 178.X.X.18 and icmp tcpdump: data link type LINUX_SLL2 tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes 17:01:05.137546 eth4-0 In ifindex 4 78:19:f7:XX:XX:XX ethertype IPv4 (0x0800), length 104: (tos 0x0, ttl 56, id 64325, offset 0, flags [DF], proto ICMP (1), length 84) 178.X.X.18 > 89.X.X2: ICMP echo request, id 33972, seq 6, length 64 17:01:05.137546 wan0 In ifindex 8 78:19:f7:XX:XX:XX ethertype IPv4 (0x0800), length 104: (tos 0x0, ttl 56, id 64325, offset 0, flags [DF], proto ICMP (1), length 84) 178.X.X.18 > 89.X.X2: ICMP echo request, id 33972, seq 6, length 64 17:01:05.137546 wan0.650 In ifindex 12 78:19:f7:XX:XX:XX ethertype IPv4 (0x0800), length 104: (tos 0x0, ttl 56, id 64325, offset 0, flags [DF], proto ICMP (1), length 84) 178.X.X.18 > 89.X.X2: ICMP echo request, id 33972, seq 6, length 64 17:01:05.137600 lo In ifindex 1 00:00:00:00:00:00 ethertype IPv4 (0x0800), length 104: (tos 0x0, ttl 55, id 64325, offset 0, flags [DF], proto ICMP (1), length 84) 178.X.X.18 > 89.X.X2: ICMP echo request, id 33972, seq 6, length 64 17:01:05.137614 lo In ifindex 1 00:00:00:00:00:00 ethertype IPv4 (0x0800), length 104: (tos 0x0, ttl 54, id 64325, offset 0, flags [DF], proto ICMP (1), length 84) 178.X.X.18 > 89.X.X2: ICMP echo request, id 33972, seq 6, length 64 17:01:05.137622 lo In ifindex 1 00:00:00:00:00:00 ethertype IPv4 (0x0800), length 104: (tos 0x0, ttl 53, id 64325, offset 0, flags [DF], proto ICMP (1), length 84) 178.X.X.18 > 89.X.X2: ICMP echo request, id 33972, seq 6, length 64 .. 17:01:05.137989 lo In ifindex 1 00:00:00:00:00:00 ethertype IPv4 (0x0800), length 104: (tos 0x0, ttl 2, id 64325, offset 0, flags [DF], proto ICMP (1), length 84) 178.38.116.18 > 89.X.X2: ICMP echo
