Hi Ali & Jeffrey & all
This is an excellent discussion and I am glad it was brought up as I would
like to chime in.
I believe AFAIK and a guess the lack of clarity in RFC 7438 which was
initially designed for flood & learn with common control plane and data
plane and then a few years later the NVO3 added EVPN separate control plane
and data plane with integrating RFC 7432 EVPN service model.
However, the service model in I would call legacy VXLAN specification RFC
7348 based on flood and learn data plane design was only explicitly defined
vlan based and was never updated to incorporate the other two service
models vlan aware bundle and vlan bundle service model.
Unfortunately this has lead to vendor interoperability issues over the
years.
This has led to industry interoperability issues with VXLAN where for
example Cisco adopted the original RFC 7348 vlan based service model where
Juniper adopted the more flexible vlan aware bundle service model. Similar
vendor adoption issues happend with RFC 9135 inter subnet forwarding where
for example Cisco adopted symmetric IRB which was touted as more efficient
where most other vendors adopted asymmetric IRB and only very late in the
game now have the vendors have budged to support the other IRB
implementations.
This has led to significant interoperability issues with VXLAN.
In the DC MPLS / SR-MPLS in the DC has been able to get around the
interoperability issues as MPLS EVPN RFC 7432 supports all 3 service models
so is a benefit to steer away from VXLAN for mix vendor interoperability.
However RFC 9135 symmetric IRB now is supported by most all vendors is not
an issue.
Here is a breakdown of the service model support issue between Cisco &
Juniper as example and I think if we update RFC 7348 much early on I think
that will help with VXLAN vendors supporting all 3 service models for
interoperability which is the overall goal of the IETF and standardization.
It is not too late and I support updating VXLAN RFC 7348 to incorporate
explicitly the other two service models
The statement describes the shift from the original flood-and-learn VXLAN
(RFC 7348, not 7438) to the more scalable EVPN-VXLAN architecture,
detailing specific differences in service models that cause
interoperability problems between vendors like Cisco and Juniper
.
The evolution from flood-and-learn to EVPN
- *Original VXLAN (Flood and Learn, RFC 7348):*The initial specification
for VXLAN relied on a data-plane flood-and-learn mechanism. A VXLAN Tunnel
End-Point (VTEP) would flood unknown unicast, broadcast, and multicast
(BUM) traffic to other VTEPs within the same VXLAN network identifier
(VNI), which required the use of IP multicast in the underlay network. This
approach proved inefficient and unscalable for large data center
environments.
-
- *EVPN Control Plane (RFC 7432 and 8365):* The adoption of Ethernet
VPN (EVPN), defined in RFC 7432 and extended for Network Virtualization
Overlay (NVO3) in RFC 8365, introduced a standards-based control plane
using Multiprotocol BGP (MP-BGP) for VXLAN overlays. This change shifted
end-host and VTEP discovery from a flooded data plane to a more efficient
and scalable control plane.
EVPN service models and vendor differences
RFC 7432 describes three main service models for how VLANs are handled
within an EVPN instance (EVI), which is a single virtual service:
- *VLAN-based service (EVI per VLAN):* Each VLAN is mapped to its own
unique EVPN instance (and L2 VNI). This was the original design philosophy
and is explicitly supported by Cisco.
- *VLAN-bundle service (EVI for multiple VLANs):* Multiple VLANs are
mapped to a single EVI and share a single L2 VNI. The VLAN tag is stripped
at the ingress VTEP and re-added at the egress VTEP.
- *VLAN-aware bundle service (Single EVI, multiple BDs):* A single EVPN
instance (EVI) carries multiple VLANs, with each VLAN being a separate
broadcast domain (BD). This model uses a single MAC VRF for multiple bridge
tables, one per VLAN tag. It provides better scalability than the
VLAN-based model because it requires fewer BGP EVPN routes for multiple
VLANs. This model is supported by Juniper and other vendors.
The interoperability gap:
The core of the interoperability issue lies in how different vendors
implement and prioritize these service models, particularly the VLAN-based
and VLAN-aware bundle services.
- *Cisco's implementation:* In many Cisco VXLAN implementations
(particularly with symmetric Integrated Routing and Bridging, IRB), the
native mode of operation is often a VLAN-based approach, mapping one VLAN
to one VNI.
- *Juniper's implementation:* Juniper often emphasizes the VLAN-aware
bundle service model, which uses a single EVPN instance to handle multiple
VLANs efficiently.
- *The incompatibility:* This difference in implementation creates
incompatibilities when connecting fabrics from these two vendors. For
example, asymmetric IRB (used by Juniper and Arista) and symmetric IRB
(used by Cisco) are fundamentally different approaches to handling Layer 3
traffic and cannot be directly interconnected. Even when both vendors
support the VLAN-based model, achieving interoperability may require
significant manual configuration ("knob pulling and pushing") to align
their behaviors.
Kind Regards
Gyan
On Thu, Sep 4, 2025 at 11:18 AM Ali Sajassi (sajassi) <sajassi=
[email protected]> wrote:
> Hi Wei,
>
> The traffic of the bundle doesn’t make sense IMO for L2 traffic because L2
> traffic is in context of <BD, EVI>. However, for L3 traffic (symmetric
> IRB), one can define bundle to mean EVI in which case the traffic for EVI
> (IP-VRF) is identified by MPLS label-2 or VNI-2 (from RT-2).
>
> Cheers,
> Ali
>
> *From: *Wei Wang <[email protected]>
> *Date: *Thursday, September 4, 2025 at 12:04 AM
> *To: *Ali Sajassi (sajassi) <[email protected]>, Jeffrey
> (Zhaohui) Zhang <[email protected]>, Selvakumar Sivaraj
> <[email protected]>, Aijun Wang <[email protected]>, 'BESS' <
> [email protected]>
> *Subject: *[bess] Re: VXLAN encapsulation question
>
> Hi Ali,
>
> As you said "In data-plane VNI always identifies a BD for both VLAN-aware
> bundle service, VLAN-based service, and VLAN bundle service."
> I'm curious about that if we wan to distinguish the traffic of a *bundle* on
> the data plane, how could we achieve it?
>
> Best Regards,
> Wei
>
> Original
> ------------------------------
> From: Ali Sajassi (sajassi) <[email protected]>
> Date: 2025年9月4日 13:01
> To: Jeffrey (Zhaohui) Zhang <[email protected]>, Selvakumar
> Sivaraj <[email protected]>, Aijun Wang <[email protected]>,
> 'BESS' <[email protected]>
> Subject: [bess] Re: VXLAN encapsulation question
>
> Hi Jeffrey,
>
> Please see my comments inline ...
>
> *From: *Jeffrey (Zhaohui) Zhang <[email protected]>
> *Date: *Wednesday, September 3, 2025 at 12:20 PM
> *To: *Ali Sajassi (sajassi) <[email protected]>, Selvakumar Sivaraj <
> [email protected]>, Aijun Wang <[email protected]>, 'BESS' <
> [email protected]>
> *Subject: *RE: [bess] Re: VXLAN encapsulation question
>
> Hi Ali,
>
>
>
> Thanks for your clarification.
>
> Ali> You’re welcome
>
> Juniper Business Use Only
>
> *From:* Ali Sajassi (sajassi) <[email protected]>
> *Sent:* Wednesday, September 3, 2025 12:54 PM
> *To:* Selvakumar Sivaraj <[email protected]>; Aijun Wang <
> [email protected]>; Jeffrey (Zhaohui) Zhang <[email protected]>;
> 'BESS' <[email protected]>
> *Subject:* Re: [bess] Re: VXLAN encapsulation question
>
>
>
> *[External Email. Be cautious of content]*
>
>
>
> Hi Jeffrey,
>
>
>
> RFC8365 was written to be consistent with RFC7348 because it is adding
> EVPN control plane to the VxLAN data plane defined by NVO3. So, if you look
> at RFC7348, it talks about inner VLAN tag handling and how it should NOT be
> sent unless configured otherwise.
>
>
>
> Zzh> I forgot about the RFC7348 base; however, only the EVPN talks about
> those different service models, so if I want to match the encapsulation to
> the service models I want to get the answers from RFC8365 (which currently
> only provides partial answers).
>
> Ali> It is implicitly there but if we want to make it explicit, then we
> can simply change the sentence in section 5.1
> from
>
>
>
> This mode of operation in [*RFC7348
> <https://datatracker.ietf.org/doc/html/rfc7348>*] maps to VLAN-Based Service
> in [*RFC7432 <https://datatracker.ietf.org/doc/html/rfc7432>*], where a tenant
> VID gets mapped to an EVI.
>
> to:
> This mode of operation in [*RFC7348
> <https://datatracker.ietf.org/doc/html/rfc7348>*] can map to VLAN-Based
> Service or VLAN-Aware Bundle Service in [*RFC7432
> <https://datatracker.ietf.org/doc/html/rfc7432>*], where a tenant VID
> gets mapped to a BD.
>
>
>
> Section 5.1.3 of RFC8365 describes how to construct EVPN BGP routes for
> VLAN-aware bundle service (3rd para). Using this section along with section
> 5.1 (VxLAN encapsulation), you will have your answer about VLAN-aware
> bundle service - i.e., data-plane encapsulation is like VLAN-based service
> similar to that of RFC7432. In data-plane VNI always identifies a BD for
> both VLAN-aware bundle service, VLAN-based service, and VLAN bundle
> service. The difference among them is in control plane route advertisement.
> So, to answer your questions specifically:
>
>
>
> Zzh> My question is not about what to use to identify the BT/BD (let’s
> forget about that one). I just want to get a simple and clear answer from
> the RFC whether/when the VLAN is included in the encapsulated frame.
>
> 1. For VLAN-aware bundle service, VLAN tag is not included (by
> default) unless configured otherwise (for passing .1P bits transparently or
> avoiding tag removal/addition). Even when you include the tag, the tag is
> NOT used for forwarding. It is the VNI that identifies the BD!
> 2. For VLAN bundle service, as stated in the section 5.1 of RFC8365,
> the tag is included in the encapsulated frame but again it is NOT used for
> forwarding decision. The tag just gets passed transparently to the host per
> RFC7432 procedure.
>
> Zzh> Section 5.1 is specifically about encapsulation, and it specifically
> mention vlan-based and vlan-bundle, so it is natural/important to include
> vlan-aware bundle as well. BTW – does the option of including the vlan tag
> (e.g. for passing .1P bits or avoiding tag removal/audition) apply to
> vlan-based as well?
>
> Ali> We can change the sentence in section 5.1 as I suggested above. And
> the option of carrying tag for .1p bits or avoiding tag/removal/addition
> also applies to VLAN-based service (but as previously mentioned and as it
> is stated in both RFC 7348 and RFC8365, the default mode is to strip it).
>
> Zzh> For the VLAN-Bundle service, the reason I have the question is that
> the text mentions “an **option** of including an inner VLAN tag in the
> encapsulated frame” – I wanted to confirm that for the VLAN-Bundle that is
> mandatory.
>
> Ali> The draft says “VxLAN provides an option of including inner VLAN tag
> …”, which means RFC7348 provides an option …. The option that is provided
> by RFC7348 MUST be used if we want to provide VLAN-bundle service. If you
> want to create an errata so that we can incorporate these clarifications, I
> am fine with it.
>
> Thanks for reviewing these documents and paying careful attention.
>
> Cheers,
> Ali
>
>
>
> Frankly, I don’t see any issues with the existing specifications in the
> RFC8365. Can we wordsmith it a bit better? Sure, but it will be just
> wordsmithing.
>
>
>
> Zzh> I hope the above explains why I had these questions (again, please
> forget about issue how to identify the BT/BD).
>
> Zzh> Thanks.
>
> Zzh> Jeffrey
>
>
>
> Cheers,
>
> Ali
>
>
>
>
>
> *From: *Selvakumar Sivaraj <*[email protected]
> <[email protected]>*>
> *Date: *Monday, September 1, 2025 at 4:43 AM
> *To: *Aijun Wang <*[email protected] <[email protected]>*>,
> 'Jeffrey (Zhaohui) Zhang' <*[email protected]
> <[email protected]>*>, 'BESS' <*[email protected]
> <[email protected]>*>
> *Subject: *[bess] Re: VXLAN encapsulation question
>
> >1. What about vlan-aware bundle? Is the vlan tag included in the
> encapsulated frame? There is no text for that.
> >2. For vlan bundle service, is the vlan tag optional or mandated in the
> encapsulated frame?
>
>
>
> In the context of VLAN-aware and VLAN-based services, the VLAN tag is
> optional. A scenario where the VLAN tag is carried is when the .1P bits
> need to be preserved end to end.
>
>
>
> >We are also wondering how to implement the "VLAN-aware bundle service"
> in the data plane.
>
>
>
> Data plane sees no difference between the three service types w.r.to
> identifying the bridge domain. In all services, the bridge table is
> identified using the VNI.
>
>
>
> >for LSI bundle service.
>
> For the scenarios captured in the document, did you explore double VxLAN
> encapsulation instead of protocol changes and custom modifications to VxLAN
> header? VxLAN packets that are received CE traverses the MAN network and
> reaches PE which then encapsulates the received frame in VxLAN
> encapsulation and sends it to other PE’'s. Receiving PE decapsulates the
> outer VxLAN header and forwards it to the CE.
>
>
>
> Thanks,
>
> Selvakumar
>
>
>
>
>
> Juniper Business Use Only
>
> *From: *Aijun Wang <*[email protected]
> <[email protected]>*>
> *Date: *Monday, September 1, 2025 at 12:27
> *To: *'Jeffrey (Zhaohui) Zhang' <*[email protected]
> <[email protected]>*>, 'BESS' <*[email protected]
> <[email protected]>*>
> *Subject: *[bess] Re: VXLAN encapsulation question
>
> [External Email. Be cautious of content]
>
>
> Hi, Jeffrey:
>
> There are several occurrences for "VLAN-aware bundle service" in RFC 8365,
> but they focus mainly on the control plane advertisements, not the
> encapsulation data plane.
> We are also wondering how to implement the "VLAN-aware bundle service" in
> the data plane.
>
> If there is none, should we consider to standardize it?
> This is also the reason of extension that is described in
> *https://urldefense.com/v3/__https://datatracker.ietf.org/doc/draft-wang-bess-l3-accessible-evpn/__;!!NEt6yMaO-gk!HLiopGnAVNJstDNrf3QL0LvigYTp0pOKbQQ29koU6KZ9azStYREbdVDklnsJEMmG521a3MoBQMaewqipn62Q63Tb$
> <https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/draft-wang-bess-l3-accessible-evpn/__;!!NEt6yMaO-gk!HLiopGnAVNJstDNrf3QL0LvigYTp0pOKbQQ29koU6KZ9azStYREbdVDklnsJEMmG521a3MoBQMaewqipn62Q63Tb$>*
> for LSI bundle service.
>
> Aijun Wang
> China Telecom
>
> -----Original Message-----
> From: *[email protected] <[email protected]>*
> [*mailto:[email protected]
> <[email protected]>*] On Behalf Of Jeffrey (Zhaohui) Zhang
> Sent: Friday, August 29, 2025 4:58 AM
> To: 'BESS' <*[email protected] <[email protected]>*>
> Subject: [bess] VXLAN encapsulation question
>
> Hi,
>
> RFC8365 says:
>
> VXLAN encapsulation is based on UDP, with an 8-byte header following
> the UDP header. VXLAN provides a 24-bit VNI, which typically
> provides a one-to-one mapping to the tenant VID, as described in
> [RFC7348]. In this scenario, the ingress VTEP does not include an
> inner VLAN tag on the encapsulated frame, and the egress VTEP
> discards the frames with an inner VLAN tag. This mode of operation
> in [RFC7348] maps to VLAN-Based Service in [RFC7432], where a tenant
> VID gets mapped to an EVI.
>
> VXLAN also provides an option of including an inner VLAN tag in the
> encapsulated frame, if explicitly configured at the VTEP. This mode
> of operation can map to VLAN Bundle Service in [RFC7432] because all
> the tenant's tagged frames map to a single bridge table / MAC-VRF,
> and the inner VLAN tag is not used for lookup by the disposition PE
> when performing VXLAN decapsulation as described in Section 6 of
> [RFC7348].
>
> I have two questions:
>
> 1. What about vlan-aware bundle? Is the vlan tag included in the
> encapsulated frame? There is no text for that.
> 2. For vlan bundle service, is the vlan tag optional or mandated in the
> encapsulated frame?
>
> I also wonder if "and the inner VLAN tag is not used for lookup by the
> disposition PE
> when performing VXLAN decapsulation as described in Section 6 of
> [RFC7348]" should be part of the reason (the text follows "because
> ..."), or the following text is better?
>
> ... This mode
> of operation can map to VLAN Bundle Service in [RFC7432] because all
> the tenant's tagged frames map to a single bridge table / MAC-VRF,
> *though* the inner VLAN tag is not used for lookup by the disposition PE
> when performing VXLAN decapsulation as described in Section 6 of
> [RFC7348].
>
> i.e., s/and/though/
> In fact, I wonder if "because" should also be changed to "where".
>
> Thanks.
> Jeffrey
>
> Juniper Business Use Only
>
> _______________________________________________
> BESS mailing list -- *[email protected] <[email protected]>*
> To unsubscribe send an email to *[email protected]
> <[email protected]>*
>
> _______________________________________________
> BESS mailing list -- *[email protected] <[email protected]>*
> To unsubscribe send an email to *[email protected]
> <[email protected]>*
>
> _______________________________________________
> BESS mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
>
_______________________________________________
BESS mailing list -- [email protected]
To unsubscribe send an email to [email protected]
