Hi everyone,
We are currently trying to expand our L2 VLAN-based CloudStack environment to
include EVPN VXLAN. We've run into a problem that we can't seem to solve…
But right from the start: We are using two Juniper Virtual Chassis, each with
two Juniper EX4650 switches. The KVM hosts (Ubuntu 24.04 with FRRouting 10.6.1)
use an LACP bond (bond1) for guest traffic. We have included the script needed
to create the bridges. The bridge is created correctly; we see traffic from the
VM’s "vnet" interface passing through the bridge and then exiting the bond
encapsulated in VXLAN. However, the traffic simply does not seem to arrive on
the other side. It doesn't matter whether the KVM hosts are connected to the
same switch chassis. If the KVM hosts communicate directly with each other via
BGP, everything works without any problems.
We noticed that the ARP table within the VM is not populated correctly. That's
strange, because both the switches and the KVM hosts have filled their ARP
tables with all the MAC addresses. But even if we add the missing MAC addresses
of the VMs on both sides, they still cannot communicate with each other. That
doesn't seem to be the problem, but perhaps it's a consequence of the actual
problem?
Our current FRR configuration looks like this:
!
frr version 10.6.0
frr defaults traditional
hostname kvm-h1
log syslog informational
service integrated-vtysh-config
!
ip prefix-list LOOPBACKS seq 10 permit 10.0.0.0/8 le 32
!
interface bond1
ip ospf area 0.0.0.51
ip ospf network point-to-point
no ipv6 nd suppress-ra
exit
!
interface lo
ip address 10.66.200.3/32
exit
!
router bgp 65203
bgp router-id 10.66.200.3
no bgp ebgp-requires-policy
no bgp default ipv4-unicast
no bgp network import-check
neighbor uplinks peer-group
neighbor uplinks remote-as external
neighbor uplinks ebgp-multihop
neighbor uplinks update-source lo
neighbor 10.65.200.250 peer-group uplinks
neighbor 10.65.200.250 description SWITCH1
!
address-family l2vpn evpn
neighbor uplinks activate
advertise-all-vni
vni 10003168
route-target import 100:100
route-target export 100:100
proxy-arp
exit-vni
advertise-svi-ip
exit-address-family
exit
!
router ospf
ospf router-id 10.66.200.3
redistribute connected route-map OSPF_EXPORT
exit
!
route-map OSPF_EXPORT permit 10
match ip address prefix-list LOOPBACKS
exit
!
end
Our test VNI is 10003168. It contains two VMs, each on a different KVM host,
which in turn are connected to two different virtual switches.
The "bond1" interface is used, which is defined in the Netplan as follows:
bonds:
bond1:
mtu: "9000"
interfaces:
- ens2f1np1
- eno3np1
addresses:
- "10.65.200.5/31"
parameters:
mode: "802.3ad"
lacp-rate: "slow"
transmit-hash-policy: "layer3+4"
This is how our juniper configuration looks like:
set protocols evpn no-core-isolation
set protocols evpn encapsulation vxlan
set protocols evpn default-gateway no-gateway-community
set protocols evpn duplicate-mac-detection detection-threshold 5
set protocols evpn duplicate-mac-detection detection-window 180
set protocols evpn duplicate-mac-detection auto-recovery-time 15
set protocols evpn multicast-mode ingress-replication
set protocols evpn extended-vni-list 4004070
set protocols evpn extended-vni-list 10003168
set routing-options router-id 10.65.200.250
set switch-options vrf-target target:100:100
set switch-options route-distinguisher 10.65.200.250:100
Session to EX4650 VC2
set protocols bgp group BGP-SW-to-SW multihop ttl 2
set protocols bgp group BGP-SW-to-SW multihop no-nexthop-change
set protocols bgp group BGP-SW-to-SW family inet unicast
set protocols bgp group BGP-SW-to-SW family evpn signaling
set protocols bgp group BGP-SW-to-SW neighbor 10.65.100.250 description AS65100
set protocols bgp group BGP-SW-to-SW neighbor 10.65.100.250 local-address
10.65.200.250
set protocols bgp group BGP-SW-to-SW neighbor 10.65.100.250 peer-as 65100
set protocols bgp group BGP-SW-to-KVM multihop ttl 2
set protocols bgp group BGP-SW-to-KVM multihop no-nexthop-change
set protocols bgp group BGP-SW-to-KVM family inet unicast
set protocols bgp group BGP-SW-to-KVM family evpn signaling
set protocols bgp group BGP-SW-to-KVM neighbor 10.66.200.3 description "AS65203"
set protocols bgp group BGP-SW-to-KVM neighbor 10.66.200.3 local-address
10.65.200.250
set protocols bgp group BGP-SW-to-KVM neighbor 10.66.200.3 peer-as 65203
set protocols bgp local-as 65200
Does anyone have experience with EVPN-VXLAN and Juniper EX4650 switches? It’s
probably a really silly problem… but we just can’t figure it out.
Thanks!
