We followed this blog post by Wido: https://wido.blog/2023/05/evpnbgpvxlan-between-junos-and-frr-with-cumulus/
From your perspective, is there any reason not to configure multiple VNI/VLANs in a single routing instance? Is one L3 interface per VLAN absolutely required? We don't want to route between VLANs. We'll test your configuration and let you know. đ Von: JĂșlĂus ĂĂłr Bess <[email protected]> Datum: Freitag, 29. Mai 2026 um 13:34 An: [email protected] <[email protected]> Betreff: Re: AW: AW: AW: Re: VXLAN with EVPN Problem Hi, Can you provide a link to the blogpost? Regarding the mac-vrf's... The Juniper technician might be unfamiliar with vlan-based service-type. The most commonly used is vlan-bundle and vlan-aware service-types. You rarely see vlan-based setup instructions for Juniper. It's not a convenient setup in regards to configuration on the juniper side but it's not much different from creating a new vlan. You create the mac-vrf and the vlan in that mac-vrf. The main downside is how you map those vlans to interfaces, if you have many interfaces and the switch is the edge. You need to use "flexable-vlan-tagging" and "encapsulation extended-vlan-bridge", create sub interfaces (unit) with the vlan tag and map each sub-interface to that mac-vrf. Example config: routing-instance mvrf-10100: instance-type mac-vrf; protocols { evpn { encapsulation vxlan; default-gateway do-not-advertise; } } vtep-source-interface lo0.0; service-type vlan-based; route-distinguisher 10.1.1.1:100; vrf-target target:65001:10100; vlans { v100 { vlan-id 100; interface ae10.100; interface ae11.100; l3-interface irb.100; vxlan { vni 10100; } } } routing-instance mvrf-10105: instance-type mac-vrf; protocols { evpn { encapsulation vxlan; default-gateway do-not-advertise; } } vtep-source-interface lo0.0; service-type vlan-based; route-distinguisher 10.1.1.1:105; vrf-target target:65001:10105; vlans { v105 { vlan-id 105; interface ae10.105; interface ae11.105; l3-interface irb.105; vxlan { vni 10105; } } } trimmed layer 3 routing-instance: instance-type vrf; route-distinguisher 10.1.1.1:300001; interface irb.100 interface irb.105 ae10: flexible-vlan-tagging; encapsulation extended-vlan-bridge; unit 100{ vlan-id 100; } unit 105{ vlan-id 105; } ae11: flexible-vlan-tagging; encapsulation extended-vlan-bridge; unit 100{ vlan-id 100; } unit 105{ vlan-id 105; } There may be another more convenient way of configuring this, I don't know, I did not find another way to map vlan tags between physical interfaces and mac-vrf's. If you do, please share it. :) On 5/29/26 7:00 AM, Julian Sluimann wrote: > Thanks for the heads-up! Just to clarify: we want a 1:1 mapping. One > VLAN should correspond to exactly one VXLAN VNI. > > The hint regarding Tag 0 was actually given to us by Juniper Support. > According to Juniper, the correct mapping from VNI to VLAN cannot take > place this way. > So, are you saying that FRR is doing the correct export here? What > surprises us is that both the RT and the VNI label are applied > correctly. This seems to be an issue between JunOS and FRR. > > Creating a routing instance for each VNI is not practicable for us. We > adopted the Route Targets from a blog post by Wido. > > Thanks for your help! > > *Von: *JĂșlĂus ĂĂłr Bess <[email protected]> > *Datum: *Donnerstag, 28. Mai 2026 um 17:30 > *An: *[email protected] <[email protected]> > *Betreff: *Re: AW: AW: Re: VXLAN with EVPN Problem > > FRR supports vlan-based service-type. That's the reason that type-2 > routes don't include the VNI. The part after the colon in the RT's > refers to the VNI. > > I haven't used vlan-aware bundle service-type myself but you might be > able to setup vlan-based service-type routing instances for each VNI > and import/export from there into your vlan-aware bundled instance. > Not very nice, but might get the job done. In that case route-targets > are automatically created on FRR side, so maybe that's a little > consolation. ;) > > On 5/28/26 3:02 PM, Julian Sluimann wrote: > > Hi JĂșlĂus, > > Yes, we intentionally use a single VLAN-aware bundle EVPN > instance. We were advised to keep it simple in one shared setup. > We do not need inter-VNI/VLAN routing, so splitting into multiple > EVPN instances would just add unnecessary complexity for our use > case. Our plan is to use VNIs as simple l2 domains - like VLAN. We > don't have to split this into different routing instances, because > they don't need to be isolated from one another. > > ip -d link show dev vxlan4004055 > 19: vxlan4004055: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > noqueue master brvx-4004055 state UNKNOWN mode DEFAULT group > default qlen 1000 > link/ether 36:d9:fe:f8:e0:6b brd ff:ff:ff:ff:ff:ff promiscuity > 1 allmulti 1 minmtu 68 maxmtu 65535 > info: Using default fan map value (33) > vxlan id 4004055 local 10.66.200.2 srcport 0 0 dstport 4789 > nolearning ttl auto ageing 300 udpcsum noudp6zerocsumtx > noudp6zerocsumrx > bridge_slave state forwarding priority 32 cost 100 hairpin off > guard off root_block off fastleave off learning on flood on > port_id 0x8001 port_no 0x1 designated_port 32769 designated_cost 0 > designated_bridge 8000.92:49:a7:74:a0:f6 designated_root > 8000.92:49:a7:74:a0:f6 hold_timer 0.00 message_age_timer > 0.00 forward_delay_timer 0.00 topology_change_ack 0 > config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 > mcast_fast_leave off mcast_flood on bcast_flood on > mcast_to_unicast off neigh_suppress off group_fwd_mask 0 > group_fwd_mask_str 0x0 vlan_tunnel off isolated off locked off > addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 > gso_max_segs 65535 tso_max_size 65536 tso_max_segs 65535 > gro_max_size 65536 > > *Julian Sluimann* > > IT-Service > > Phone Icon > 0256498996-30 > Email Icon > [email protected] > > > epnox_logo.png > Website Icon > epnox.de > Location Icon > Stadtlohner Str. 6, 48691 Vreden > > epnox GmbH Stadtlohner StraĂe 6 48691 Vreden > GeschĂ€ftsfĂŒhrer: Gerd Gevering | Christian Meiners | Nils Waning | > Ust.-Nummer: DE351345026 > St.Nr.: 301/5813/2061 | HRB 20199 | zertifiziert nach: ISO 27001 > > *Bitte prĂŒfen Sie vorher, ob ein Ausdruck dieser E-Mail wirklich > notwendig ist.* > > Diese E-Mail enthĂ€lt vertrauliche und/oder rechtlich geschĂŒtzte > Informationen. Wenn Sie nicht der richtige Adressat sind oder > diese E-Mail irrtĂŒmlich erhalten haben, informieren Sie bitte > sofort den Absender und vernichten Sie diese Mail. Das unerlaubte > Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht > gestattet. This e-mail may contain confidential and/or privileged > information. If you are not the intended recipient (or have > received this e-mail in error) please notify the sender > immediately and destroy this e-mail. Any unauthorised copying, > disclosure or distribution of the material in this e-mail is > strictly forbidden. > > *Von: *JĂșlĂus ĂĂłr Bess <[email protected]> > *Datum: *Donnerstag, 28. Mai 2026 um 16:30 > *An: *[email protected] <[email protected]> > *Betreff: *Re: AW: Re: VXLAN with EVPN Problem > > Hi Julian, > > Why do you mix the VNI's into one RT? Are you trying to do vlan-bundle > service type on the FRR side? > > Also, can you show ip -d link show dev <vxlan dev> > > On 5/28/26 11:53 AM, Julian Sluimann wrote: > > Hey everyone, > > > > Yesterday we reviewed the whole situation together with Juniper > support. According to them, the overall setup and configuration on > the Juniper side looks correct. However, they suspect that the > issue is related to how FRR advertises the EVPN routes. > > > > Specifically, the Type-2 routes received from FRR do not include > the VNI in the route itself (see output below â EthTag/VNI is > shown as "0"). The switch appears to recognize the VNI as a route > label and learns the MAC addresses, but due to the missing VNI > field it does not properly associate the MAC entries with the EVPN > MAC-IP table. This would also explain why communication works > locally on the same switch, but not across EVPN/VXLAN between the > servers. > > > > The interesting part is that the VNI is shown correctly for the > MAC address "48:a9:8a:ef:75:73". However, this device is not > connected via VXLAN directly â it is a MikroTik board connected > through a regular Layer-2 VLAN which is mapped into the VNI on the > EX4650 side. Communication between the VMs and the MikroTik board > is only partially working (75% packet loss). Only the two VM MAC > addresses ("02:04:02:64:00:01" and "02:04:02:64:00:02") learned > from FRR are missing the VNI information in the Type-2 routes: > > > > show bgp l2vpn evpn route type 2 > > BGP table version is 4, local router ID is 10.66.200.3 > > Network Next Hop Metric LocPrf Weight Path > > Extended Community > > Route Distinguisher: 10.65.200.240:100 > > *> [2]:[4004055]:[48]:[48:a9:8a:ef:75:73] RD 10.65.200.240:100 > > 10.65.200.240 0 65400 i > > RT:100:100 ET:8 > > > > Route Distinguisher: 10.66.200.2:3 > > *> [2]:[0]:[48]:[02:04:02:64:00:01] RD 10.66.200.2:3 > > 10.66.200.2 0 65400 65202 i > > RT:100:100 ET:8 > > > > Route Distinguisher: 10.66.200.3:3 > > *> [2]:[0]:[48]:[02:04:02:64:00:02] RD 10.66.200.3:3 > > 10.66.200.3 32768 i > > ET:8 RT:100:100 > > > > Juniperâs statement was: > > > > âBased on our findings, it appears that the servers are not > sending their Type-2 routes with the VNI included in the route, > which may be preventing the creation of entries in the EVPN MAC-IP > table. In addition, the next-hop entry seems to be missing from > the PFE.â > > > > We also discussed this with another engineer who has extensive > experience with Juniper and EVPN deployments. He also does not see > a fundamental issue on the Juniper side, but unfortunately he has > no practical experience with FRR in this specific scenario. > > > > Maybe some of you have experience with FRR? Do you use it in a > similar way to us? Maybe we just missing a small detail in the config: > > > > ! > > frr version 10.6.0 > > frr defaults traditional > > hostname cs-kvm-st-cl1-03 > > log syslog informational > > log file /var/log/frr/debug.log debugging > > service integrated-vtysh-config > > ! > > ip prefix-list LOOPBACKS seq 10 permit 10.0.0.0/8 le 32 > > ! > > interface bond1 > > ip ospf area 0.0.0.51 > > ip ospf network point-to-point > > no ipv6 nd suppress-ra > > exit > > ! > > interface lo > > ip address 10.66.200.3/32 > > exit > > ! > > router bgp 65203 > > bgp router-id 10.66.200.3 > > no bgp ebgp-requires-policy > > no bgp default ipv4-unicast > > no bgp network import-check > > neighbor uplinks peer-group > > neighbor uplinks remote-as external > > neighbor uplinks ebgp-multihop > > neighbor uplinks update-source lo > > neighbor 10.65.200.240 peer-group uplinks > > ! > > address-family l2vpn evpn > > neighbor uplinks activate > > neighbor uplinks next-hop-self > > advertise-all-vni > > advertise ipv4 unicast > > vni 4004070 > > route-target import 100:100 > > route-target export 100:100 > > proxy-arp > > exit-vni > > vni 4004055 > > route-target import 100:100 > > route-target export 100:100 > > advertise-svi-ip > > proxy-arp > > exit-vni > > vni 10030864 > > route-target import 100:100 > > route-target export 100:100 > > proxy-arp > > exit-vni > > advertise-svi-ip > > exit-address-family > > exit > > ! > > router ospf > > ospf router-id 10.66.200.3 > > redistribute connected route-map OSPF_EXPORT > > exit > > ! > > route-map OSPF_EXPORT permit 10 > > match ip address prefix-list LOOPBACKS > > exit > > ! > > end > > > > Thanks for your help! > > > > Von: Julian Sluimann <[email protected]> > > Datum: Dienstag, 19. Mai 2026 um 10:48 > > An: [email protected] <[email protected]> > > Cc: Wido den Hollander <[email protected]> > > Betreff: RE: Re: VXLAN with EVPN Problem > > > > Hey Wido! > > > > Thanks for reply! > > > >> I must say I have never tried with a JunOS VC, but why would > you also? > >> This is a full L3 setup, correct? Why add the VC of JunOS? > > We are currently using two virtual chassis because it was an > existing setup with running redundant L2 connections. > > We still need these connections. Our plan was to migrate to a L3 > environment now. In a new setup we would use four single devices too. > > Right now this is not possible because of production systems. > > > >> Which MACs, where? Who is your gateway? > > We are seeing all mac-addresses on the right sides in evpn > database. These are published to evpn fine. > > > >> Why not use this for the BGP session? > > We are using a bond because of the virual chassis on the switch > side. We wanted to use 2x25G connection. We are using OSPF to > learn Loopback IPs of devices. > > BGP connection via Loopback is the reason for one single bgp > session between switch and kvm host. > > > >> Did you set a vtep-source under switch-options to lo0.0? > > Yes, of course we set vtep Interface to loopback. > > > > set switch-options vtep-source-interface lo0.0 > > > > Can you tell me something about your import/export rules or add > a snippet too? > > Right now we are not using any policies. > > > > Our next steps are: We will try to run bgp sessions via > interface addresses without OSPF. We will try with a single spare > device without VC. > > We thought about using iBGP too but decided to use eBGP because > of easier debugging. > > > > Thank you for your help! > > > > On 2026/05/19 04:14:35 Wido den Hollander via users wrote: > >> > >> Op 18-05-2026 om 11:25 schreef Julian Sluimann: > >>> Hi everyone, > >>> > >>> We are currently trying to expand our L2 VLAN-based CloudStack > environment to include EVPN VXLAN. We've run into a problem that > we can't seem to solve⊠> >>> > >>> But right from the start: We are using two Juniper Virtual > Chassis, each with two Juniper EX4650 switches. The KVM hosts > (Ubuntu 24.04 with FRRouting 10.6.1) use an LACP bond (bond1) for > guest traffic. We have included the script needed to create the > bridges. The bridge is created correctly; we see traffic from the > VMâs "vnet" interface passing through the bridge and then exiting > the bond encapsulated in VXLAN. However, the traffic simply does > not seem to arrive on the other side. It doesn't matter whether > the KVM hosts are connected to the same switch chassis. If the KVM > hosts communicate directly with each other via BGP, everything > works without any problems. > >>> > >> I do see some problems with FRR <> JunOS communicating with > BGP+EVPN, > >> but it can work. > >> > >> I must say I have never tried with a JunOS VC, but why would > you also? > >> This is a full L3 setup, correct? Why add the VC of JunOS? > >> > >>> We noticed that the ARP table within the VM is not populated > correctly. That's strange, because both the switches and the KVM > hosts have filled their ARP tables with all the MAC addresses. But > even if we add the missing MAC addresses of the VMs on both sides, > they still cannot communicate with each other. That doesn't seem > to be the problem, but perhaps it's a consequence of the actual > problem? > >> Which MACs, where? Who is your gateway? > >> > >> (see more below) > >> > >>> Our current FRR configuration looks like this: > >>> > >>> ! > >>> frr version 10.6.0 > >>> frr defaults traditional > >>> hostname kvm-h1 > >>> log syslog informational > >>> service integrated-vtysh-config > >>> ! > >>> ip prefix-list LOOPBACKS seq 10 permit 10.0.0.0/8 le 32 > >>> ! > >>> interface bond1 > >>> ip ospf area 0.0.0.51 > >>> ip ospf network point-to-point > >>> no ipv6 nd suppress-ra > >>> exit > >>> ! > >>> interface lo > >>> ip address 10.66.200.3/32 > >>> exit > >>> ! > >>> router bgp 65203 > >>> bgp router-id 10.66.200.3 > >>> no bgp ebgp-requires-policy > >>> no bgp default ipv4-unicast > >>> no bgp network import-check > >>> neighbor uplinks peer-group > >>> neighbor uplinks remote-as external > >>> neighbor uplinks ebgp-multihop > >>> neighbor uplinks update-source lo > >> Correct? How does the EX learn this loopback address of the KVM > host? > >> Why not use the address of the interface for the uplink? > >> > >> And why a bond and not two seperate BGP sessions? > >> > >>> neighbor 10.65.200.250 peer-group uplinks > >>> neighbor 10.65.200.250 description SWITCH1 > >>> ! > >>> address-family l2vpn evpn > >>> neighbor uplinks activate > >>> advertise-all-vni > >>> vni 10003168 > >>> route-target import 100:100 > >>> route-target export 100:100 > >>> proxy-arp > >>> exit-vni > >>> advertise-svi-ip > >>> exit-address-family > >>> exit > >>> ! > >>> router ospf > >>> ospf router-id 10.66.200.3 > >>> redistribute connected route-map OSPF_EXPORT > >>> exit > >>> ! > >>> route-map OSPF_EXPORT permit 10 > >>> match ip address prefix-list LOOPBACKS > >>> exit > >>> ! > >>> end > >>> > >>> Our test VNI is 10003168. It contains two VMs, each on a > different KVM host, which in turn are connected to two different > virtual switches. > >>> > >>> The "bond1" interface is used, which is defined in the Netplan > as follows: > >>> > >>> bonds: > >>> bond1: > >>> mtu: "9000" > >>> interfaces: > >>> - ens2f1np1 > >>> - eno3np1 > >>> addresses: > >>> - "10.65.200.5/31" > >> Why not use this for the BGP session? > >> > >>> parameters: > >>> mode: "802.3ad" > >>> lacp-rate: "slow" > >>> transmit-hash-policy: "layer3+4" > >>> > >>> This is how our juniper configuration looks like: > >>> > >>> set protocols evpn no-core-isolation > >>> set protocols evpn encapsulation vxlan > >>> set protocols evpn default-gateway no-gateway-community > >>> set protocols evpn duplicate-mac-detection detection-threshold 5 > >>> set protocols evpn duplicate-mac-detection detection-window 180 > >>> set protocols evpn duplicate-mac-detection auto-recovery-time 15 > >>> set protocols evpn multicast-mode ingress-replication > >>> set protocols evpn extended-vni-list 4004070 > >>> set protocols evpn extended-vni-list 10003168 > >>> > >>> set routing-options router-id 10.65.200.250 > >>> set switch-options vrf-target target:100:100 > >>> set switch-options route-distinguisher 10.65.200.250:100 > >> Did you set a vtep-source under switch-options to lo0.0? > >> > >>> Session to EX4650 VC2 > >>> > >>> set protocols bgp group BGP-SW-to-SW multihop ttl 2 > >>> set protocols bgp group BGP-SW-to-SW multihop no-nexthop-change > >>> set protocols bgp group BGP-SW-to-SW family inet unicast > >>> set protocols bgp group BGP-SW-to-SW family evpn signaling > >>> set protocols bgp group BGP-SW-to-SW neighbor 10.65.100.250 > description AS65100 > >>> set protocols bgp group BGP-SW-to-SW neighbor 10.65.100.250 > local-address 10.65.200.250 > >>> set protocols bgp group BGP-SW-to-SW neighbor 10.65.100.250 > peer-as 65100 > >>> > >>> set protocols bgp group BGP-SW-to-KVM multihop ttl 2 > >>> set protocols bgp group BGP-SW-to-KVM multihop no-nexthop-change > >>> set protocols bgp group BGP-SW-to-KVM family inet unicast > >>> set protocols bgp group BGP-SW-to-KVM family evpn signaling > >>> set protocols bgp group BGP-SW-to-KVM neighbor 10.66.200.3 > description "AS65203" > >>> set protocols bgp group BGP-SW-to-KVM neighbor 10.66.200.3 > local-address 10.65.200.250 > >>> set protocols bgp group BGP-SW-to-KVM neighbor 10.66.200.3 > peer-as 65203 > >> A snippet of the config on a QFX5120 > >> > >> > >> group compute { > >> type external; > >> multihop { > >> no-nexthop-change; > >> } > >> accept-remote-nexthop; > >> import compute-in; > >> family inet { > >> unicast { > >> extended-nexthop; > >> } > >> } > >> family inet6 { > >> unicast; > >> } > >> family evpn { > >> signaling; > >> } > >> export compute-out; > >> neighbor 2a01:XXX:2:180::7 { > >> description compute0; > >> local-address 2a01:XXX:2:180::6; > >> peer-as 64650; > >> } > >> } > >> > >> Here we peer over IPv6, so yes, slightly different. > >> > >> > >>>> set protocols bgp local-as 65200 > >>> Does anyone have experience with EVPN-VXLAN and Juniper EX4650 > switches? Itâs probably a really silly problem⊠but we just canât > figure it out. > >> As said, the QFX5120 gave me some troubles as well, but it works! > >> > >> Wido > >> > >>> Thanks! > >>> > >> >
