Hi Jeffrey, Thanks a lot for your professional comments. Please see my response inline with [Tiger]
发件人: Jeffrey Haas <[email protected]> 日期: 星期五, 2026年9月18日 05:31 收件人: Tiger Xu <[email protected]> 抄送: Jeff Tantsura <[email protected]>, Robert Raszuk <[email protected]>, Stephane Litkowski (slitkows) <[email protected]>, idr <[email protected]>, BESS <[email protected]> 主题: Re: [Idr] WG adoption of draft-xu-idr-fare-08 (8/18/206 to 9/1/2026) Tiger (and other FARE authors), I've made some of these comments at the IETF mic before. When I discuss the BGP link-bandwidth community with people learning BGP, I start by focusing on the fact that the WECMP values are derived from the received bandwidth values at a given BGP speaker. The implementations largely flatten the much larger floating point value into a small index range ECMP value in the line cards. The remaining part of my discussion is the same signaling mechanism, link-bandwidth, may run at many places with different agendas. Our most boring example, the original use case, is simply biasing the ECMP for multiple BGP routes, typically received from an external AS. A common second use case is that when link-bw is used by the routes used for resolving BGP next hops, we often are interested in the cumulative bandwidth at a given node since it's the available bandwidth to the destination next hop. In that case, we need to "do math" to handle the cumulative operation. Exactly what form that math takes will depend on the operator's desires. The result is simply used to bias the ECMP of service routes resolving over that ECMP. Another use case, primarily the one addressed in your draft, is the computation of the minimal bandwidth. In cases where the traffic to a destination may consist of heavy "elephant flows" that need to know the *least* bandwidth available on the path to the next hop, especially when it's more than one hop away. (A degenerate case are other bandwidth reservation schemes that may partition a link or path among multiple use cases.) [Tiger] Your observation is absolutely correct! In each of these cases, the link-bandwidth community in one or both of its transitivity behaviors (usually transitive, overlapping why Juniper's implementation was transitive prior to RFC 10005) is sufficient to signal the capacity and permit the receiving BGP speakers to correctly do the ECMP. However, what is required in such cases is careful operational management of each scenario at a given ECMP layer (service, underlay, etc.) to get the expected result. The scenarios tended to be limited based on either the automatic "math" infrastructure the implementations had, or available options in the policy language. To the above point, the existing link-bandwidth "just works". However, the semantic is overloaded. In some scenarios, particularly where the underlay routes may want cumulative bandwidth, and overlay or egress routes may want minimum bandwidth, operational care to not let the bandwidth use cases cross is needed. [Tiger] Your observation is also correct. As I mentioned previously on the mailing list, the existing link-bandwidth attribute works fine for three-stage CLOS networks by using the “math” logic originally described in the FARE draft. Of course, the DMZ use case draft has recently added the same “math” logic. However, for five-stage CLOS networks, it becomes very difficult to leverage the existing link-bandwidth attribute to achieve the “math” logic and therefore the expected WECMP result described in the FARE draft. In fact, some network vendors have attempted to reuse the existing link-bandwidth attribute for five-stage CLOS networks but ultimately failed. Not only does this approach require operators to reconfigure the policy when the number of links between switches is adjusted during network capacity changes, but it is also incapable of achieving the expected WECMP result. As I've said previously, an additional community that helps to segregate the use cases at the different scenario layers could be helpful. However, since we're at adoption time, it'd be helpful as part of your case to do the following: - Recognize "this can be done using link-bw" (the dispute about who created the use case first aside - see the prior discussion for where this matter current stands) - Discuss the scenarios where clarity of community can ease deployment considerations. (Removing some of the need for "operational care") - And, most importantly, discuss how the FARE path-bw community interacts with the existing link-bw community. This is needed for interop no matter what. [Tiger] Your suggestions above are constructive! Clarification text corresponding to those suggestions, as well as other concerns on which the WG has reached consensus, will be added to the future revision of the WG draft. This latter point certainly should be in your draft, and obviously is an excellent candidate for presenting the scenarios at the upcoming IETF meeting. [Tiger] Unfortunately, I will not be able to attend the upcoming IETF meeting due to a US visa issue. Thank you again for your professional and constructive comments and suggestions. Best regards, Tiger -- Jeff On Sep 7, 2026, at 09:31, Tiger Xu <[email protected]> wrote: Hi Jeff, Sorry for the late reply since I just found this email in my junk folder. Let me try to clarify the logic of FARE as follows. 发件人: Jeff Tantsura <[email protected]> 日期: 星期一, 2026年9月7日 11:09 收件人: Robert Raszuk <[email protected]> 抄送: Stephane Litkowski (slitkows) <[email protected]>, idr <[email protected]> 主题: [Idr] Re: WG adoption of draft-xu-idr-fare-08 (8/18/206 to 9/1/2026) Hi Robert, If you were to overwrite remotely received BW by locally configured one, you would lose the “cumulative” nature of the proposal. It accumulates the value rather than overwriting it, making it working in RFC7938-like fabrics. If not all paths received for a prefix have BW, advertising “incomplete” set would be ambiguous. Common policy would be: If none present: - don’t advertise ∑BW (safest, since lack of BW is most likely misconfiguration/bug) or (local policy) - set default (locally configured) BW (defats the purpose, since it overwrites the previously accumulated value) All good points. And worth discussing in the wg. in line with what Stephane is saying and how I see this evolving: to separate what is encoded from how it is processed - FARE's single-session three-stage MIN-then-SUM recurrence does not require a new Extended Community. Given identical multipath membership and local policy, it is a profile of RFC 10005 and eBGP-DMZ: Section 4.2 gives per-path MIN(local-link-bandwidth, received-bandwidth), Section 4.3 by default sums it across the selected multipath set - the arithmetic RFC 10005 Section 3.4 delegates there. No new wire information. [Tiger] FARE does not require a single session per adjacent pair. Instead, it is fine to run multiple sessions per pair (e.g., one per physical link). Admittedly, the existing link-bandwidth extended community could be abused as a container for path bandwidth to realize the path-bandwidth-aware WECMP logic described in the FARE draft. However, see below... Only the five-stage case carries two differently scoped values; RFC 10005 Sections 3.1 and 4 already provides two communities on one prefix. [Tiger] Details matter here. Please review Section 7.1 (Inconsistent Deployment) alongside Sections 3.1 and 3.2 of RFC10005 to see how these two EC types should be used according to the specification. Publish eBGP-DMZ, it meets all IDR/BESS criteria - widely deployed, large number of interoperating implementations. Publish additional use cases as separate drafts, building on top of existing encodings, that complement it rather than creating ambiguity. [Tiger] The FARE protocol has more than 8 implementations and has been widely deployed (see https://datatracker.ietf.org/meeting/126/materials/slides-126-idr-sessa-1-5-1-fully-adaptive-routing-ethernet-fare-00). The FARE draft already thoroughly describes the path-bandwidth-aware WECMP logic for both three-stage and five-stage CLOS networks. This was introduced more than a year before the eBGP-DMZ draft duplicated the same logic, which only addresses the three-stage case with minimal elaboration (see https://mailarchive.ietf.org/arch/msg/idr/nzjkS2675LChF5WqrBmfUfpukMM/). To avoid ambiguity between the semantics and intended usages of path bandwidth versus link bandwidth, and to deliver comprehensive, suitable solutions for both 3-stage and 5-stage CLOS networks, I suggest that the eBGP-DMZ draft strictly limit itself to the existing use cases described in Section 3—i.e., the scenarios for which the link-bandwidth attribute was originally designed—rather than duplicating functionality already covered elsewhere. This approach would both enhance the quality of the draft and uphold professional ethics. BTW, based on previous discussions on the same draft (see https://mailarchive.ietf.org/arch/msg/idr/I3KPpGUuyouHp7Nca4qnEYwI_H4/), it would be inappropriate to advance eBGP-DMZ as a Standards Track RFC—particularly within the BESS WG. Best regards, Tiger On Sep 6, 2026, at 16:46, Robert Raszuk <[email protected]> wrote: Hi Jeff, > (as a safety measure - most implementation would not advertise BW if not all > contributing routes have it attached) I am not following this "safety measure". What reception of a path with link bw extended community would have to do with advertising any bw - which clearly is taken from local interface(s) configuration. > nothing prevent an implementation to choose the lowest value rather than a > sum of all values. Well the receiver will see some value in the extended community. Such value can be either a: - bw of a link best path was received from - as you say cumulative bw across all paths eligible and installed in RIB for multipath - as you say min bw across all paths eligible and installed in RIB for multipath Today there is no way to include in the advertisement any indicator which one of those three is being advertised so that would need to be configured by hand on a per peer basis. IMO if we have a real use case for all three they should be assigned different extended communities to avoid configuration mismatch and mistakes. And what happens when some peers advertise different meanings of the bw ? Also the behaviour when building hash on the receiving router would be quite different - especially when we are considering the reception of the min bw. The problem is that if we just limit the advertisement to min bw (and that is also actually a question to fare authors) we can not construct the has to accurately create the weighted load balancing. Best, R. On Sun, Sep 6, 2026 at 12:31 AM Jeff Tantsura <[email protected]<mailto:[email protected]>> wrote: Hi Robert, Your question was specific enough, my response is specific to cumulative bw. This is exactly what is deployed in one of the hyperscalers and most of AIDCs. Common use is to advertise the cumulative weight of all next-hops the route has been learnt from (as a safety measure - most implementation would not advertise BW if not all contributing routes have it attached), nothing prevent an implementation to chose the lowest value rather than a sum of all values. Thanks, Jeff On Sep 4, 2026, at 03:57, Robert Raszuk <[email protected]<mailto:[email protected]>> wrote: Hi Jeff, I know there are many implementations of link bandwidth extended community. Maybe my question was not articulated very clearly. I was specifically asking which shipping implementations support cumulative bw and minimum bw advertisements among ECMP links. But even if there is not many of them now it can be added so not trying to justify -fare progress based on those missing. The only justification to -fare progressing (with hopefully all AI stuff removed from the draft) would be finding a case where it is beneficial to advertise say more then one bw parameter for a group of ECMP ebgp links. For example - I want to know the min bw of all links + cumulative bw of all links. That today I am afraid is not possible with link bandwidth extended community. Cheers, Robert On Fri, Sep 4, 2026 at 3:05 AM Jeff Tantsura <[email protected]<mailto:[email protected]>> wrote: Robert, There’s at least 4 interworking implementations of draft-ietf-bess-ebgp-dmz. Cumulus, FRR, EOS, IOS-* , possibly more. It is deployed in hyperscale networks as well as in AI clusters with 10.000+ BGP speakers. Stephane - let’s take care of wiki. Thanks, Jeff On Aug 31, 2026, at 02:54, Robert Raszuk <[email protected]<mailto:[email protected]>> wrote: HI, I indeed see cumulative bw, but do not see minimum bw. Btw I was commenting on RFC10005 not a BESS WG draft ... Honestly such cumulative or minimum should be described IMO in IDR RFC10005 as it is about what to encode. > (sigma bw of all paths and minimum) as implementations support it already. Which implementations support it ? Is there draft-ietf-bess-ebgp-dmz implementation report wiki ? And for RFC10005 ( draft-ietf-idr-link-bandwidth) I do not see any implementation report on https://wiki.ietf.org/group/idr/implementations Thx, R. On Mon, Aug 31, 2026 at 11:41 AM Stephane Litkowski (slitkows) <[email protected]<mailto:[email protected]>> wrote: Hi Robert, If you look at draft-ietf-bess-ebgp-dmz which details how LBW extcom is used today, it already defines both cases (sigma bw of all paths and minimum) as implementations support it already. From: Robert Raszuk <[email protected]<mailto:[email protected]>> Sent: Monday, August 31, 2026 11:28 AM To: Stephane Litkowski (slitkows) <[email protected]<mailto:[email protected]>> Cc: idr <[email protected]<mailto:[email protected]>> Subject: Re: [Idr] Re: WG adoption of draft-xu-idr-fare-08 (8/18/206 to 9/1/2026) Hi Stephane, In respect to RFC10005 I do see a difference. Today per RFC10005 we propagate bw of the interface over which best path was received. However when you have say 32 or 64 paths it very well may get down to rtr_id best path selection step and advertised bw may be of that interface. This draft is asking to advertise the minimum bw of all paths. Is it useful or not - I am not sure. I would actually prefer to see it advertising sigma bw of all paths. Now could we advertise it using the existing Link Bandwidth Extended Community - of course. One page informational draft could propose that. - - - Now as I previously pointed out, fare draft still is trying to sell itself as a way to use vanilla IP networks to construct real AI training clusters. That's wrong .. very wrong .. Yes - it is very attractive to this community, but does not make it technically sound. It does not even fall under the "poor man's AI fabric" in my books. Cheers, Robert On Mon, Aug 31, 2026 at 10:09 AM Stephane Litkowski (slitkows) <[email protected]<mailto:[email protected]>> wrote: Hi WG, I oppose to the adoption of this document as WG item. 1. Assumptions about RFC10005 are wrong (likely outdated): * RFC10005 supports transitive community so the restriction mentioned does not apply * The draft mentions that link BW is to carry external link BW but the actual definition from RFC has a wider scope: “The Link Bandwidth Extended Community is defined as a BGP extended community that carries the bandwidth information of a router (represented by BGP next hop) that is connecting to a remote network. “. * It says that we can’t attach more that one LBW CT to a route which is true, but doesn’t talk about the use case which may require more than one. 1. I don’t like the EC to be called “Path Bandwidth”, by definition, the existing RFC10005 community is already a path bandwidth as it is attached to a BGP path. A BGP speaker advertising a bandwidth community to its neighbor, tell to its neighbor about the BW available if this neighbor uses the path advertised from this BGP speaker. As demonstrated in some other email thread, the use case described is already achieved for a long time by some implementation using the existing community (Shraddha is mentioning it as well) . BTW the new community proposed as the same weaknesses as the existing one so does not bring any value. Coming to Shraddha’s point about multiple level of loadbalancing, I don’t think this is what the draft is talking about. The use case is still the plain hop-by-hop BGP fabric. Considering a case where we have for instance a BGP service route which NH is resolved over a BGP underlay route, there is no restriction today that would prevent attaching a LBW com to the service route and another one to the BGP underlay route so each level can have some different UCMP behavior. I’m still wondering reading the draft what is the problem to be solved and why we can’t solve it using current mechanisms. Until this is fully clear, I don’t see how we can adopt the draft. Brgds, Stephane From: Hegde, Shraddha <[email protected]<mailto:[email protected]>> Sent: Thursday, August 27, 2026 11:01 AM To: Zhangqingliang <[email protected]<mailto:[email protected]>>; Susan Hares <[email protected]<mailto:[email protected]>>; idr@ietf. org <[email protected]<mailto:[email protected]>> Subject: [Idr] Re: WG adoption of draft-xu-idr-fare-08 (8/18/206 to 9/1/2026) I support the WG adoption of the draft draft-xu-idr-fare I am not aware of any IPR on this. I believe that ability to send multiple bandwidth communities is useful. In a traditional MPLS VPN network, there is a BGP overlay and IGP underlay. The BGP carried link bandwidth is used to load-balance traffic based on external bandwidth availability and the IGP bandwidth based on internal network bandwidth availability so its possible to apply two levels of bandwidth based load-balancing. In a DC network BGP plays the role of underlay transport. Many a times the service prefixes are injected into the fabric in DC. Carrying two different bandwidth values is useful to appropriately balance the traffic based on external and internal bandwidth availability. draft-xu-idr-fare proposes path bandwidth community which is a good starting point for the WG to work on. However, we should deliberate on how the RFC 10005 link bandwidth community and the path bandwidth community interact. RFC 10005 is very generic, and it is possible to advertise, minimum bandwidth or additive bandwidth or any other form of bandwidth calculation when a nexthop change is done. Which means most of the capabilities path bandwidth proposes to do is already available with link bandwidth community and might have been deployed already by operators. When both link bandwidth and path bandwidth communities are carried in a prefix, which one carries what information is something that needs to be specified in a backward compatible way given that RFC 10005 has been implemented by many vendors and widely deployed. Rgds Shraddha ________________________________ From: Zhangqingliang <[email protected]<mailto:[email protected]>> Sent: 27 August 2026 12:15 To: Susan Hares <[email protected]<mailto:[email protected]>>; idr@ietf. org <[email protected]<mailto:[email protected]>> Subject: [Idr] Re: WG adoption of draft-xu-idr-fare-08 (8/18/206 to 9/1/2026) Hi Sue, As a contributor, I support the WG adoption of this draft and am not aware of any relevant IPR. Best Regards, Qingliang Zhang From: Susan Hares <[email protected]<mailto:[email protected]>> Date: Tuesday, August 18, 2026 at 3:47 AM To: idr@ietf. org <[email protected]<mailto:[email protected]>> Subject: [Idr] WG adoption of draft-xu-idr-fare-08 (8/18/206 to 9/1/2026) This begins a 2-week WG Adoption call for draft-xu-fare-08.txt from 8/18 /2026 to 9/1/2026. The authors and contributors should reply to this email with an IPR statement. The WG may wish to consider the following comments: 1) Is the use case described in draft-xu-fare-08 found AI training clusters in data centers? 2) Does the Path Bandwidth Extended Community provide Reliable bandwidth allocation for the 3-stage and 5-stage CLOS environments? 3) Should this community be both transitive and non-transitive. Cheerily, Sue Hares [EXTERNAL] ------------------------------------------------------------------------------------------------------------------------------------- 本邮件及其附件含有新华三集团的保密信息，仅限于发送给上面地址中列出的个人或群组。 禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。 如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！ This e-mail and its attachments contain confidential information from New H3C, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it! _______________________________________________ Idr mailing list -- [email protected]<mailto:[email protected]> To unsubscribe send an email to [email protected]<mailto:[email protected]> _______________________________________________ Idr mailing list -- [email protected]<mailto:[email protected]> To unsubscribe send an email to [email protected]<mailto:[email protected]> _______________________________________________ Idr mailing list -- [email protected] To unsubscribe send an email to [email protected]
_______________________________________________ BESS mailing list -- [email protected] To unsubscribe send an email to [email protected]
