Re: HE.net and BGP Communities
I think the key difference here is that I really just wanted HE to treat my routes no differently no matter how they learned them. I wanted them to apply normal BGP routing rules to them.. that is, pick the path with the shortest AS path. >From a strictly technical basis, its silly to prefer a path of AS6939-AS-A4043 to a path of AS6969‐AS4043.I understand the business reasons of wanting to send traffic down circuits where you get paid, but how many intermediate ASNs do there need to be in the middle before it makes sense to ignore the business reasons for doing so?I guess once you get paid for pushing the traffic one doesn't really care, but at some point one has to look at the network efficiencies here. In this case, we were purchasing a DSL aggregation service from another regional provider after our DSL customer base reached the point where it no longer made sense for us to maintain our own infrastructure. What we really needed was a peering connection with them but they were technically unable to make this happen, so it ended up being transit which we found out after the agreement was signed. I vaguely remember them giving some reason why they couldn't do a no export on our routes inside their network (maybe running multiple networks under different ASNs due to acquisitions or some technical shortcomings of their network, don't remember for sure). But one way or another they ended up announcing our routes to HE and couldn't or wouldn't stop doing so. This basically bit us in the rear with HE which is why I was griping. HE has such a big footprint that that connection attracted a large percentage of our overall traffic. I would have been happy with any of several common community strings that people use.Set localpref. Don't export. And so on. Just send us traffic literally any other way other than this one. Or less traffic. Ideally, setting the localpref to the same value as a peering customer would have been ideal, because then the path length would have been taken as the selection criteria. I would have been fine if no-export came along for the ride as a side effect. What ended up happening instead is that the only way we could fix this is that we did the evil things that you discussed and split all of our prefixes in half and announced them everywhere else both ways.The regional ISP for whatever reason didn't learn the smaller prefixes so the peering worked fine (I suspect they may not have been taking a full table) And, since we now appeared as a customer of HE we ended up getting transit from them for free across our IX link, although in realty the amount of extra traffic was minimal (except during a couple of network events) . I really would have rather been able to tell Hurricane to just ignore this route or treat it as a peer or something like that. This whole service has been disconnected for some time now, so it doesn't really matter much to me other than it still frustrates me that we had to go down the other path to fix this when a commonly implemented community string may have eliminated the need to do so. On Mon, Jul 25, 2022, 4:51 PM Tony Wicks wrote: > > > > I do understand the reasoning behind preferring customer routes. > > However in the case where a customer of a customer also connects to > > you directly via peering doesn't it make sense to prefer the direct > > connection? or at least not prefer the customer learned routes. > > So from my experience of working at transit providers over more years than > I care to contemplate I can assure you what may seem to make sense as a > customer does not necessarily translate to how IP routing works. IP has no > concept of Customer or Peer it is simply designed to hand the packet to a > valid next hop as determined by policies. As such routes are normally > divided into customer, peer or further upstream transit if you are not one > of the tier 1 providers. A peer provides you no income, a customer (a > customer of a customer is largely the same thing as being a direct > customer). Take the example of the customer buying a transit service on a > 95th percentile basis. So as a transit provider I get paid based on how > much traffic I hand to that port(s) and in turn I provide connectivity to > all my peers, customers and upstream transits all over the world. I can't > do this for free as then I can't pay for my network and I am a company not > a charity. Customer X advertises his lets say a /22 to me, all good. But > then customer X advertises his /22 and some disaggregated /24's to a local > peering exchange that I am also connected to. If I do not both prioritise > customer X's customer port routes AND drop any more specific routes learnt > from customer X then I will end up handing all customer X's outgoing > traffic to them over the peering instead of the revenue generating port. I > have seen customers do this both through innocent and malicious intent. > Sure, there are a lot of complex poli
RE: HE.net and BGP Communities
> > I do understand the reasoning behind preferring customer routes. > However in the case where a customer of a customer also connects to > you directly via peering doesn't it make sense to prefer the direct > connection? or at least not prefer the customer learned routes. So from my experience of working at transit providers over more years than I care to contemplate I can assure you what may seem to make sense as a customer does not necessarily translate to how IP routing works. IP has no concept of Customer or Peer it is simply designed to hand the packet to a valid next hop as determined by policies. As such routes are normally divided into customer, peer or further upstream transit if you are not one of the tier 1 providers. A peer provides you no income, a customer (a customer of a customer is largely the same thing as being a direct customer). Take the example of the customer buying a transit service on a 95th percentile basis. So as a transit provider I get paid based on how much traffic I hand to that port(s) and in turn I provide connectivity to all my peers, customers and upstream transits all over the world. I can't do this for free as then I can't pay for my network and I am a company not a charity. Customer X advertises his lets say a /22 to me, all good. But then customer X advertises his /22 and some disaggregated /24's to a local peering exchange that I am also connected to. If I do not both prioritise customer X's customer port routes AND drop any more specific routes learnt from customer X then I will end up handing all customer X's outgoing traffic to them over the peering instead of the revenue generating port. I have seen customers do this both through innocent and malicious intent. Sure, there are a lot of complex policies that I might apply to accept local traffic in one area and hand other traffic via the transit port but why on earth would I do that and likely cause all sorts of other potential routing issues while reducing the revenue I am entitled to?
Re: HE.net and BGP Communities
most networks prefer customers over peers. a few networks don't, some large. if they changed, customers would scream at them about the newly overloaded customer circuit. we are not required to like this :) randy
Re: HE.net and BGP Communities
On Mon, 25 Jul 2022 at 16:23, Forrest Christian (List Account) wrote: > > I do understand the reasoning behind preferring customer routes. > However in the case where a customer of a customer also > connects to you directly via peering doesn't it make sense to prefer > the direct connection? or at least not prefer the customer learned routes. No, I don't think it makes sense at all. Not preferring customer routes over peering routes would mean: - less revenue for HE - less revenue for the HE customer (your transit provider) - risk of upsetting the customer for the previous reason (less revenue) - for your transit provider it would mean they would provide a transit service with less redundancy (see below), less ingress-paths (see below), and more complicated configuration (and therefore more complex troubleshooting) The only difference is that HE doesn't provide BGP communities for your transit provider for TE. *But even if HE would provide THE communities*, it's not in the best interest of your transit provider to actually pass those BGP communities to HE or configure them for you, because it would mean less traffic on revenue generating links, so it would not make sense from a business perspective. It would also likely require manual configuration of policies, which can be difficult, undesirable or not possible at all, depending on the amount of automation in that network. > I don't remember why, but we couldn't have the transit provider not announce > our routes toward HE And why would they send for your routes specifically or pass through BGP communities to HE, when they are already refusing to withdraw announcements to HE? Businesses don't usually go the extra mile to make everything worse for them (revenue, configuration and support complexity, redundancy, performance due to amount of ingress paths). HE is special in this regards for 2 reasons: - HE doesn't provide TE communities for customers (which is bad and a showstopper for some) - but more importantly, HE has an open peering policy so they actual peer with you in the first place; otherwise we wouldn't have this discussion There are a few technical aspects that to consider: Once HE's best path is a peer route, as opposed to a customer route (which is what you are asking), then those routes would not be announced by HE to its peers anymore (after all, they don't exchange peer routes with peers). So you'd be worse off in some aspects, because your ingress traffic wouldn't be attracted by HE's (vast) peering network anymore. Also if you are single-homed to this transit-provider, and this transit-provider also temporarily single-homes to HE (maybe for maintenance one 1 of 2 transits), you'd be disconnected from pretty much the rest of the internet (that doesn't have HE transit), until you shutdown your HE peering. So less ingress paths, less redundancy, etc. There are good business and technical reasons for preferring customer routes. Many networks don't even peer with customers of customers, also for good reasons. I don't think it makes any sense to consider a shorter as-path over peering when it also comes from a customer (with a longer as-path). - lukas
RE: HE.net and BGP Communities
> Google has let you down, 6730 (sunrise) is not 6939 (HE) ;) > Oops I should check thing better at the end of the day .. My bad
Re: HE.net and BGP Communities
On Mon, 25 Jul 2022 at 17:19, Brian Turnbow via NANOG wrote: > > Hi, > > >I do understand the reasoning behind preferring customer routes. However > >in the case where a customer of a customer also connects to you directly via > >peering doesn't it make sense to prefer the direct connection? or at >least > >not prefer the customer learned routes. > > Business not technical. Fill up the bandwidth and they will buy more!! > > >In our situation, we were buying transit, heavily prepended, from a provider > >on a tiny circuit. The purpose of the transit was related to another > >service we were acquiring from that provider and wasn't about the transit, > >but >the transit was needed for the service to work reliably. > >Unfortunately this provider was also a HE customer and so we now had all of > >the HE traffic coming down this tiny link, since all of our other transit > >providers and >ourselves only peered with HE. > > >I don't remember why, but we couldn't have the transit provider not > >announce our routes toward HE, so we ended up doing the announce more > >specifics everywhere else thing. Which I hate doing on so many levels. > > >Thus the desire for a community to tell HE that although they learned this > >route from a customer, it is not a customer route. > > You can use communities to set the local preference with HE. > This should do the trick > 6730:0008 set local pref to 64 (lowest they have afaik) Google has let you down, 6730 (sunrise) is not 6939 (HE) ;) Afaik HE does not provide TE communities. Lukas
RE: HE.net and BGP Communities
Hi, >I do understand the reasoning behind preferring customer routes. However in >the case where a customer of a customer also connects to you directly via >peering doesn't it make sense to prefer the direct connection? or at >least >not prefer the customer learned routes. Business not technical. Fill up the bandwidth and they will buy more!! >In our situation, we were buying transit, heavily prepended, from a provider >on a tiny circuit. The purpose of the transit was related to another service >we were acquiring from that provider and wasn't about the transit, but >the >transit was needed for the service to work reliably. Unfortunately this >provider was also a HE customer and so we now had all of the HE traffic coming >down this tiny link, since all of our other transit providers and >ourselves >only peered with HE. >I don't remember why, but we couldn't have the transit provider not announce >our routes toward HE, so we ended up doing the announce more specifics >everywhere else thing. Which I hate doing on so many levels. >Thus the desire for a community to tell HE that although they learned this >route from a customer, it is not a customer route. You can use communities to set the local preference with HE. This should do the trick 6730:0008 set local pref to 64 (lowest they have afaik) Provided that your provider accepts it and propagates it. Or you can ask them to set it for you. Brian
Re: HE.net and BGP Communities
> Am 7/25/22 um 12:45 schrieb Forrest Christian (List Account) > : > > > I wish they'd add one more that turns off their "prefer routes learned from a > customer" rule. I'm having to split my blocks in half and announce them > that way to get them to send my traffic directly to me through our IX peering > session as opposed to one of my transit providers. When you buy from the lowest bidder, this what you receive. You know how to fix it. > > I'd rather they just let shortest path selection work. > >> On Sun, Jul 24, 2022, 1:43 PM Siyuan Miao wrote: >> They do have BGP communities ... but for black-hole only :-( >> >>> On Sun, Jul 24, 2022 at 9:39 PM Ryan Hamel >>> wrote: >>> Yes. >>> >>> Ryan >>> >>> -Original Message- >>> From: NANOG On Behalf Of Rubens >>> Kuhl >>> Sent: Sunday, July 24, 2022 12:36 PM >>> To: Nanog >>> Subject: HE.net and BGP Communities >>> >>> The last mention I found on NANOG about HE.net and BGP communities for >>> traffic engineering is from April 2021 and said they provided none. >>> >>> Is that still the case a year later ? >>> >>> >>> Rubens >>>
Re: HE.net and BGP Communities
I do understand the reasoning behind preferring customer routes. However in the case where a customer of a customer also connects to you directly via peering doesn't it make sense to prefer the direct connection? or at least not prefer the customer learned routes. In our situation, we were buying transit, heavily prepended, from a provider on a tiny circuit. The purpose of the transit was related to another service we were acquiring from that provider and wasn't about the transit, but the transit was needed for the service to work reliably. Unfortunately this provider was also a HE customer and so we now had all of the HE traffic coming down this tiny link, since all of our other transit providers and ourselves only peered with HE. I don't remember why, but we couldn't have the transit provider not announce our routes toward HE, so we ended up doing the announce more specifics everywhere else thing. Which I hate doing on so many levels. Thus the desire for a community to tell HE that although they learned this route from a customer, it is not a customer route. On Mon, Jul 25, 2022, 5:21 AM Jared Mauch wrote: > You always want to prefer customer routes over non customer routes as a > service provider. Of course having a robust set of communities to let > adjustments happen helps. > > Without proper tiering of routes you may see unstable routing. > > Having a standard set of customer, peer, transit set of local preferences > would go a long way. Same for geographic scope of routes, only use these on > same continent. Makes using a provider if you do something like anycast > hard if they haul you long distance. > > - Jared > > Sent via RFC1925 compliant device > > On Jul 25, 2022, at 6:49 AM, Forrest Christian (List Account) < > li...@packetflux.com> wrote: > > > I wish they'd add one more that turns off their "prefer routes learned > from a customer" rule. I'm having to split my blocks in half and announce > them that way to get them to send my traffic directly to me through our IX > peering session as opposed to one of my transit providers. > > I'd rather they just let shortest path selection work. > > On Sun, Jul 24, 2022, 1:43 PM Siyuan Miao wrote: > >> They do have BGP communities ... but for black-hole only :-( >> >> On Sun, Jul 24, 2022 at 9:39 PM Ryan Hamel >> wrote: >> >>> Yes. >>> >>> Ryan >>> >>> -Original Message- >>> From: NANOG On Behalf Of >>> Rubens Kuhl >>> Sent: Sunday, July 24, 2022 12:36 PM >>> To: Nanog >>> Subject: HE.net and BGP Communities >>> >>> The last mention I found on NANOG about HE.net and BGP communities for >>> traffic engineering is from April 2021 and said they provided none. >>> >>> Is that still the case a year later ? >>> >>> >>> Rubens >>> >>>
Re: HE.net and BGP Communities
>I wish they'd add one more that turns off their "prefer routes learned from a >customer" rule. I'm having to split my blocks in >half and announce them >that way to get them to send my traffic directly to me through our IX peering >session as opposed to >one of my transit providers. >I'd rather they just let shortest path selection work. I think this is by design so you don't end up with free inbound transit. If one of their transit customers is trying to reach your prefixes, my guess it'd make sense to offload that over IX first, although I'm not sure if that's always happens due to path selection.
Re: HE.net and BGP Communities
You always want to prefer customer routes over non customer routes as a service provider. Of course having a robust set of communities to let adjustments happen helps. Without proper tiering of routes you may see unstable routing. Having a standard set of customer, peer, transit set of local preferences would go a long way. Same for geographic scope of routes, only use these on same continent. Makes using a provider if you do something like anycast hard if they haul you long distance. - Jared Sent via RFC1925 compliant device > On Jul 25, 2022, at 6:49 AM, Forrest Christian (List Account) > wrote: > > > I wish they'd add one more that turns off their "prefer routes learned from a > customer" rule. I'm having to split my blocks in half and announce them > that way to get them to send my traffic directly to me through our IX peering > session as opposed to one of my transit providers. > > I'd rather they just let shortest path selection work. > >> On Sun, Jul 24, 2022, 1:43 PM Siyuan Miao wrote: >> They do have BGP communities ... but for black-hole only :-( >> >>> On Sun, Jul 24, 2022 at 9:39 PM Ryan Hamel >>> wrote: >>> Yes. >>> >>> Ryan >>> >>> -Original Message- >>> From: NANOG On Behalf Of Rubens >>> Kuhl >>> Sent: Sunday, July 24, 2022 12:36 PM >>> To: Nanog >>> Subject: HE.net and BGP Communities >>> >>> The last mention I found on NANOG about HE.net and BGP communities for >>> traffic engineering is from April 2021 and said they provided none. >>> >>> Is that still the case a year later ? >>> >>> >>> Rubens >>>
Re: HE.net and BGP Communities
I wish they'd add one more that turns off their "prefer routes learned from a customer" rule. I'm having to split my blocks in half and announce them that way to get them to send my traffic directly to me through our IX peering session as opposed to one of my transit providers. I'd rather they just let shortest path selection work. On Sun, Jul 24, 2022, 1:43 PM Siyuan Miao wrote: > They do have BGP communities ... but for black-hole only :-( > > On Sun, Jul 24, 2022 at 9:39 PM Ryan Hamel > wrote: > >> Yes. >> >> Ryan >> >> -Original Message- >> From: NANOG On Behalf Of >> Rubens Kuhl >> Sent: Sunday, July 24, 2022 12:36 PM >> To: Nanog >> Subject: HE.net and BGP Communities >> >> The last mention I found on NANOG about HE.net and BGP communities for >> traffic engineering is from April 2021 and said they provided none. >> >> Is that still the case a year later ? >> >> >> Rubens >> >>
Re: HE.net and BGP Communities
They do have BGP communities ... but for black-hole only :-( On Sun, Jul 24, 2022 at 9:39 PM Ryan Hamel wrote: > Yes. > > Ryan > > -Original Message- > From: NANOG On Behalf Of > Rubens Kuhl > Sent: Sunday, July 24, 2022 12:36 PM > To: Nanog > Subject: HE.net and BGP Communities > > The last mention I found on NANOG about HE.net and BGP communities for > traffic engineering is from April 2021 and said they provided none. > > Is that still the case a year later ? > > > Rubens > >
RE: HE.net and BGP Communities
Yes. Ryan -Original Message- From: NANOG On Behalf Of Rubens Kuhl Sent: Sunday, July 24, 2022 12:36 PM To: Nanog Subject: HE.net and BGP Communities The last mention I found on NANOG about HE.net and BGP communities for traffic engineering is from April 2021 and said they provided none. Is that still the case a year later ? Rubens
