Hi Júlíus,
Yes, we intentionally use a single VLAN-aware bundle EVPN instance. We were
advised to keep it simple in one shared setup. We do not need inter-VNI/VLAN
routing, so splitting into multiple EVPN instances would just add unnecessary
complexity for our use case. Our plan is to use VNIs as simple l2 domains -
like VLAN. We don't have to split this into different routing instances,
because they don't need to be isolated from one another.
ip -d link show dev vxlan4004055
19: vxlan4004055: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
master brvx-4004055 state UNKNOWN mode DEFAULT group default qlen 1000
link/ether 36:d9:fe:f8:e0:6b brd ff:ff:ff:ff:ff:ff promiscuity 1 allmulti
1 minmtu 68 maxmtu 65535
info: Using default fan map value (33)
vxlan id 4004055 local 10.66.200.2 srcport 0 0 dstport 4789 nolearning ttl
auto ageing 300 udpcsum noudp6zerocsumtx noudp6zerocsumrx
bridge_slave state forwarding priority 32 cost 100 hairpin off guard off
root_block off fastleave off learning on flood on port_id 0x8001 port_no 0x1
designated_port 32769 designated_cost 0 designated_bridge
8000.92:49:a7:74:a0:f6 designated_root 8000.92:49:a7:74:a0:f6 hold_timer
0.00 message_age_timer 0.00 forward_delay_timer 0.00 topology_change_ack
0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1
mcast_fast_leave off mcast_flood on bcast_flood on mcast_to_unicast off
neigh_suppress off group_fwd_mask 0 group_fwd_mask_str 0x0 vlan_tunnel off
isolated off locked off addrgenmode eui64 numtxqueues 1 numrxqueues 1
gso_max_size 65536 gso_max_segs 65535 tso_max_size 65536 tso_max_segs 65535
gro_max_size 65536
Julian Sluimann
IT-Service
[cid:phone-icon_004b6ca1-3c9d-4d5f-a392-9e20f385200f.png] 0256498996-30
[cid:mail-icon_0faae6df-067c-4ca9-a09f-d32cbfd051ea.png] [email protected]
[cid:epnoxlogo_45de1f2a-1d32-4b3a-8ea0-3fc3a81feb48.png]
[cid:website-icon_cd56ec0c-da12-41a5-a7c5-4e2997773e4c.png] epnox.de
[cid:location-icon_d9aaf7ae-803a-430f-b99e-11ccf4e4ccff.png] Stadtlohner
Str. 6, 48691 Vreden
epnox GmbH Stadtlohner Straße 6 48691 Vreden
Geschäftsführer: Gerd Gevering | Christian Meiners | Nils Waning | Ust.-Nummer:
DE351345026
St.Nr.: 301/5813/2061 | HRB 20199 | zertifiziert nach: ISO 27001
[cid:leaf-icon-green_4afffd01-0555-42af-9279-0b6f0b5dd129.png] Bitte prüfen Sie
vorher, ob ein Ausdruck dieser E-Mail wirklich notwendig ist.
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten
haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht
gestattet. This e-mail may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this e-mail in error)
please notify the sender immediately and destroy this e-mail. Any unauthorised
copying, disclosure or distribution of the material in this e-mail is strictly
forbidden.
Von: Júlíus Þór Bess <[email protected]>
Datum: Donnerstag, 28. Mai 2026 um 16:30
An: [email protected] <[email protected]>
Betreff: Re: AW: Re: VXLAN with EVPN Problem
Hi Julian,
Why do you mix the VNI's into one RT? Are you trying to do vlan-bundle
service type on the FRR side?
Also, can you show ip -d link show dev <vxlan dev>
On 5/28/26 11:53 AM, Julian Sluimann wrote:
> Hey everyone,
>
> Yesterday we reviewed the whole situation together with Juniper support.
> According to them, the overall setup and configuration on the Juniper side
> looks correct. However, they suspect that the issue is related to how FRR
> advertises the EVPN routes.
>
> Specifically, the Type-2 routes received from FRR do not include the VNI in
> the route itself (see output below – EthTag/VNI is shown as "0"). The switch
> appears to recognize the VNI as a route label and learns the MAC addresses,
> but due to the missing VNI field it does not properly associate the MAC
> entries with the EVPN MAC-IP table. This would also explain why communication
> works locally on the same switch, but not across EVPN/VXLAN between the
> servers.
>
> The interesting part is that the VNI is shown correctly for the MAC address
> "48:a9:8a:ef:75:73". However, this device is not connected via VXLAN directly
> — it is a MikroTik board connected through a regular Layer-2 VLAN which is
> mapped into the VNI on the EX4650 side. Communication between the VMs and the
> MikroTik board is only partially working (75% packet loss). Only the two VM
> MAC addresses ("02:04:02:64:00:01" and "02:04:02:64:00:02") learned from FRR
> are missing the VNI information in the Type-2 routes:
>
> show bgp l2vpn evpn route type 2
> BGP table version is 4, local router ID is 10.66.200.3
> Network Next Hop Metric LocPrf Weight Path
> Extended Community
> Route Distinguisher: 10.65.200.240:100
> *> [2]:[4004055]:[48]:[48:a9:8a:ef:75:73] RD 10.65.200.240:100
> 10.65.200.240 0 65400 i
> RT:100:100 ET:8
>
> Route Distinguisher: 10.66.200.2:3
> *> [2]:[0]:[48]:[02:04:02:64:00:01] RD 10.66.200.2:3
> 10.66.200.2 0 65400 65202 i
> RT:100:100 ET:8
>
> Route Distinguisher: 10.66.200.3:3
> *> [2]:[0]:[48]:[02:04:02:64:00:02] RD 10.66.200.3:3
> 10.66.200.3 32768 i
> ET:8 RT:100:100
>
> Juniper’s statement was:
>
> “Based on our findings, it appears that the servers are not sending their
> Type-2 routes with the VNI included in the route, which may be preventing the
> creation of entries in the EVPN MAC-IP table. In addition, the next-hop entry
> seems to be missing from the PFE.”
>
> We also discussed this with another engineer who has extensive experience
> with Juniper and EVPN deployments. He also does not see a fundamental issue
> on the Juniper side, but unfortunately he has no practical experience with
> FRR in this specific scenario.
>
> Maybe some of you have experience with FRR? Do you use it in a similar way to
> us? Maybe we just missing a small detail in the config:
>
> !
> frr version 10.6.0
> frr defaults traditional
> hostname cs-kvm-st-cl1-03
> log syslog informational
> log file /var/log/frr/debug.log debugging
> service integrated-vtysh-config
> !
> ip prefix-list LOOPBACKS seq 10 permit 10.0.0.0/8 le 32
> !
> interface bond1
> ip ospf area 0.0.0.51
> ip ospf network point-to-point
> no ipv6 nd suppress-ra
> exit
> !
> interface lo
> ip address 10.66.200.3/32
> exit
> !
> router bgp 65203
> bgp router-id 10.66.200.3
> no bgp ebgp-requires-policy
> no bgp default ipv4-unicast
> no bgp network import-check
> neighbor uplinks peer-group
> neighbor uplinks remote-as external
> neighbor uplinks ebgp-multihop
> neighbor uplinks update-source lo
> neighbor 10.65.200.240 peer-group uplinks
> !
> address-family l2vpn evpn
> neighbor uplinks activate
> neighbor uplinks next-hop-self
> advertise-all-vni
> advertise ipv4 unicast
> vni 4004070
> route-target import 100:100
> route-target export 100:100
> proxy-arp
> exit-vni
> vni 4004055
> route-target import 100:100
> route-target export 100:100
> advertise-svi-ip
> proxy-arp
> exit-vni
> vni 10030864
> route-target import 100:100
> route-target export 100:100
> proxy-arp
> exit-vni
> advertise-svi-ip
> exit-address-family
> exit
> !
> router ospf
> ospf router-id 10.66.200.3
> redistribute connected route-map OSPF_EXPORT
> exit
> !
> route-map OSPF_EXPORT permit 10
> match ip address prefix-list LOOPBACKS
> exit
> !
> end
>
> Thanks for your help!
>
> Von: Julian Sluimann <[email protected]>
> Datum: Dienstag, 19. Mai 2026 um 10:48
> An: [email protected] <[email protected]>
> Cc: Wido den Hollander <[email protected]>
> Betreff: RE: Re: VXLAN with EVPN Problem
>
> Hey Wido!
>
> Thanks for reply!
>
>> I must say I have never tried with a JunOS VC, but why would you also?
>> This is a full L3 setup, correct? Why add the VC of JunOS?
> We are currently using two virtual chassis because it was an existing setup
> with running redundant L2 connections.
> We still need these connections. Our plan was to migrate to a L3 environment
> now. In a new setup we would use four single devices too.
> Right now this is not possible because of production systems.
>
>> Which MACs, where? Who is your gateway?
> We are seeing all mac-addresses on the right sides in evpn database. These
> are published to evpn fine.
>
>> Why not use this for the BGP session?
> We are using a bond because of the virual chassis on the switch side. We
> wanted to use 2x25G connection. We are using OSPF to learn Loopback IPs of
> devices.
> BGP connection via Loopback is the reason for one single bgp session between
> switch and kvm host.
>
>> Did you set a vtep-source under switch-options to lo0.0?
> Yes, of course we set vtep Interface to loopback.
>
> set switch-options vtep-source-interface lo0.0
>
> Can you tell me something about your import/export rules or add a snippet too?
> Right now we are not using any policies.
>
> Our next steps are: We will try to run bgp sessions via interface addresses
> without OSPF. We will try with a single spare device without VC.
> We thought about using iBGP too but decided to use eBGP because of easier
> debugging.
>
> Thank you for your help!
>
> On 2026/05/19 04:14:35 Wido den Hollander via users wrote:
>>
>> Op 18-05-2026 om 11:25 schreef Julian Sluimann:
>>> Hi everyone,
>>>
>>> We are currently trying to expand our L2 VLAN-based CloudStack environment
>>> to include EVPN VXLAN. We've run into a problem that we can't seem to solve…
>>>
>>> But right from the start: We are using two Juniper Virtual Chassis, each
>>> with two Juniper EX4650 switches. The KVM hosts (Ubuntu 24.04 with
>>> FRRouting 10.6.1) use an LACP bond (bond1) for guest traffic. We have
>>> included the script needed to create the bridges. The bridge is created
>>> correctly; we see traffic from the VM’s "vnet" interface passing through
>>> the bridge and then exiting the bond encapsulated in VXLAN. However, the
>>> traffic simply does not seem to arrive on the other side. It doesn't matter
>>> whether the KVM hosts are connected to the same switch chassis. If the KVM
>>> hosts communicate directly with each other via BGP, everything works
>>> without any problems.
>>>
>> I do see some problems with FRR <> JunOS communicating with BGP+EVPN,
>> but it can work.
>>
>> I must say I have never tried with a JunOS VC, but why would you also?
>> This is a full L3 setup, correct? Why add the VC of JunOS?
>>
>>> We noticed that the ARP table within the VM is not populated correctly.
>>> That's strange, because both the switches and the KVM hosts have filled
>>> their ARP tables with all the MAC addresses. But even if we add the missing
>>> MAC addresses of the VMs on both sides, they still cannot communicate with
>>> each other. That doesn't seem to be the problem, but perhaps it's a
>>> consequence of the actual problem?
>> Which MACs, where? Who is your gateway?
>>
>> (see more below)
>>
>>> Our current FRR configuration looks like this:
>>>
>>> !
>>> frr version 10.6.0
>>> frr defaults traditional
>>> hostname kvm-h1
>>> log syslog informational
>>> service integrated-vtysh-config
>>> !
>>> ip prefix-list LOOPBACKS seq 10 permit 10.0.0.0/8 le 32
>>> !
>>> interface bond1
>>> ip ospf area 0.0.0.51
>>> ip ospf network point-to-point
>>> no ipv6 nd suppress-ra
>>> exit
>>> !
>>> interface lo
>>> ip address 10.66.200.3/32
>>> exit
>>> !
>>> router bgp 65203
>>> bgp router-id 10.66.200.3
>>> no bgp ebgp-requires-policy
>>> no bgp default ipv4-unicast
>>> no bgp network import-check
>>> neighbor uplinks peer-group
>>> neighbor uplinks remote-as external
>>> neighbor uplinks ebgp-multihop
>>> neighbor uplinks update-source lo
>> Correct? How does the EX learn this loopback address of the KVM host?
>> Why not use the address of the interface for the uplink?
>>
>> And why a bond and not two seperate BGP sessions?
>>
>>> neighbor 10.65.200.250 peer-group uplinks
>>> neighbor 10.65.200.250 description SWITCH1
>>> !
>>> address-family l2vpn evpn
>>> neighbor uplinks activate
>>> advertise-all-vni
>>> vni 10003168
>>> route-target import 100:100
>>> route-target export 100:100
>>> proxy-arp
>>> exit-vni
>>> advertise-svi-ip
>>> exit-address-family
>>> exit
>>> !
>>> router ospf
>>> ospf router-id 10.66.200.3
>>> redistribute connected route-map OSPF_EXPORT
>>> exit
>>> !
>>> route-map OSPF_EXPORT permit 10
>>> match ip address prefix-list LOOPBACKS
>>> exit
>>> !
>>> end
>>>
>>> Our test VNI is 10003168. It contains two VMs, each on a different KVM
>>> host, which in turn are connected to two different virtual switches.
>>>
>>> The "bond1" interface is used, which is defined in the Netplan as follows:
>>>
>>> bonds:
>>> bond1:
>>> mtu: "9000"
>>> interfaces:
>>> - ens2f1np1
>>> - eno3np1
>>> addresses:
>>> - "10.65.200.5/31"
>> Why not use this for the BGP session?
>>
>>> parameters:
>>> mode: "802.3ad"
>>> lacp-rate: "slow"
>>> transmit-hash-policy: "layer3+4"
>>>
>>> This is how our juniper configuration looks like:
>>>
>>> set protocols evpn no-core-isolation
>>> set protocols evpn encapsulation vxlan
>>> set protocols evpn default-gateway no-gateway-community
>>> set protocols evpn duplicate-mac-detection detection-threshold 5
>>> set protocols evpn duplicate-mac-detection detection-window 180
>>> set protocols evpn duplicate-mac-detection auto-recovery-time 15
>>> set protocols evpn multicast-mode ingress-replication
>>> set protocols evpn extended-vni-list 4004070
>>> set protocols evpn extended-vni-list 10003168
>>>
>>> set routing-options router-id 10.65.200.250
>>> set switch-options vrf-target target:100:100
>>> set switch-options route-distinguisher 10.65.200.250:100
>> Did you set a vtep-source under switch-options to lo0.0?
>>
>>> Session to EX4650 VC2
>>>
>>> set protocols bgp group BGP-SW-to-SW multihop ttl 2
>>> set protocols bgp group BGP-SW-to-SW multihop no-nexthop-change
>>> set protocols bgp group BGP-SW-to-SW family inet unicast
>>> set protocols bgp group BGP-SW-to-SW family evpn signaling
>>> set protocols bgp group BGP-SW-to-SW neighbor 10.65.100.250 description
>>> AS65100
>>> set protocols bgp group BGP-SW-to-SW neighbor 10.65.100.250 local-address
>>> 10.65.200.250
>>> set protocols bgp group BGP-SW-to-SW neighbor 10.65.100.250 peer-as 65100
>>>
>>> set protocols bgp group BGP-SW-to-KVM multihop ttl 2
>>> set protocols bgp group BGP-SW-to-KVM multihop no-nexthop-change
>>> set protocols bgp group BGP-SW-to-KVM family inet unicast
>>> set protocols bgp group BGP-SW-to-KVM family evpn signaling
>>> set protocols bgp group BGP-SW-to-KVM neighbor 10.66.200.3 description
>>> "AS65203"
>>> set protocols bgp group BGP-SW-to-KVM neighbor 10.66.200.3 local-address
>>> 10.65.200.250
>>> set protocols bgp group BGP-SW-to-KVM neighbor 10.66.200.3 peer-as 65203
>> A snippet of the config on a QFX5120
>>
>>
>> group compute {
>> type external;
>> multihop {
>> no-nexthop-change;
>> }
>> accept-remote-nexthop;
>> import compute-in;
>> family inet {
>> unicast {
>> extended-nexthop;
>> }
>> }
>> family inet6 {
>> unicast;
>> }
>> family evpn {
>> signaling;
>> }
>> export compute-out;
>> neighbor 2a01:XXX:2:180::7 {
>> description compute0;
>> local-address 2a01:XXX:2:180::6;
>> peer-as 64650;
>> }
>> }
>>
>> Here we peer over IPv6, so yes, slightly different.
>>
>>
>>>> set protocols bgp local-as 65200
>>> Does anyone have experience with EVPN-VXLAN and Juniper EX4650 switches?
>>> It’s probably a really silly problem… but we just can’t figure it out.
>> As said, the QFX5120 gave me some troubles as well, but it works!
>>
>> Wido
>>
>>> Thanks!
>>>
>>
