Re: BGP route hijack by AS10990

On Mon, Aug 03, 2020 at 08:57:53AM -0400, Tom Beecher wrote: > Telia made a mistake. They owned it and will endeavor to do better. What > more can be asked? Figure out how that mistake happened -- what factors led to it? Then make changes so that it can't happen again, at least not in that partic

Re: BGP route hijack by AS10990

On 3/Aug/20 17:09, Baldur Norddahl wrote: > > We suffered a series of crashes that led to JTAC recommending > disabling RPKI. We had a core dump which matches PR1332626 which is > confidential, so I have no idea what it is about. Apparently what > happened was the server running the RPKI validat

Re: BGP route hijack by AS10990

On Mon, Aug 3, 2020 at 3:54 PM Job Snijders wrote: > On Mon, Aug 03, 2020 at 02:36:25PM +0200, Alex Band wrote: > > According to the information I received from the community[1], you > > should read PR1461602 and PR1309944 before deploying. > > > > [1] https://rpki.readthedocs.io/en/latest/rpki/r

Re: BGP route hijack by AS10990

On 1/Aug/20 02:44, Rafael Possamai wrote: > To your point with regards to multiple failures combined causing an > outage, here's some basic reading on the Swiss cheese model: > https://en.wikipedia.org/wiki/Swiss_cheese_model You just reminded me of the defense's strategy in the court case agai

Re: BGP route hijack by AS10990

On 3/Aug/20 14:57, Tom Beecher wrote: > Agreed.  > > However, every time we go on this Righteous Indignation of Should Do > crusade, it would serve us well to stop and remember that in every one > of our jobs, at many points in our careers, we have been faced with a > situation where something

Re: BGP route hijack by AS10990

On Mon, Aug 03, 2020 at 02:36:25PM +0200, Alex Band wrote: > According to the information I received from the community[1], you > should read PR1461602 and PR1309944 before deploying. > > [1] https://rpki.readthedocs.io/en/latest/rpki/router-support.html My take on PR1461602 is that it can be ign

Re: BGP route hijack by AS10990

To your point with regards to multiple failures combined causing an outage, here's some basic reading on the Swiss cheese model: https://en.wikipedia.org/wiki/Swiss_cheese_model >From over here it looks like the legacy filter was a latent failure, and the >BGP automation from the downstream pe

Re: BGP route hijack by AS10990

On 3/Aug/20 14:36, Alex Band wrote: > According to the information I received from the community[1], you should > read PR1461602 and PR1309944 before deploying. The good news is the code that fixes both of those issues is shipping. Mark.

Re: BGP route hijack by AS10990

> > We can all do better. We should all do better. > Agreed. However, every time we go on this Righteous Indignation of Should Do crusade, it would serve us well to stop and remember that in every one of our jobs, at many points in our careers, we have been faced with a situation where something

Re: BGP route hijack by AS10990

> On 3 Aug 2020, at 11:04, adamv0...@netconsultings.com wrote: > >> Darrell Budic >> Sent: Sunday, August 2, 2020 6:23 PM >> >> On Jul 30, 2020, at 5:37 PM, Baldur Norddahl >> wrote: >>> >>> Telia implements RPKI filtering so the question is did it work? Were any >> affected prefixes RPKI sig

RE: BGP route hijack by AS10990

> Darrell Budic > Sent: Sunday, August 2, 2020 6:23 PM > > On Jul 30, 2020, at 5:37 PM, Baldur Norddahl > wrote: > > > > Telia implements RPKI filtering so the question is did it work? Were any > affected prefixes RPKI signed? Would any prefixes have avoided being > hijacked if RPKI signing had b

Re: BGP route hijack by AS10990

On 2/Aug/20 19:22, Darrell Budic wrote: > Oh uh, I’m getting close to getting RPKI going on my mx204s, or was until you > posted that. What’s the story there, and perhaps which junos version? None that I know if. We have it working well (RPKI + ROV) on MX204's running Junos 19.2. Curious to

Re: BGP route hijack by AS10990

On Jul 30, 2020, at 5:37 PM, Baldur Norddahl wrote: > > Telia implements RPKI filtering so the question is did it work? Were any > affected prefixes RPKI signed? Would any prefixes have avoided being hijacked > if RPKI signing had been in place? > > Regards > > Baldur - who had to turn off R

Re: BGP route hijack by AS10990

> On Aug 1, 2020, at 12:59 PM, Sabri Berisha wrote: > > - On Aug 1, 2020, at 12:50 PM, Nick Hilliard n...@foobar.org wrote: > > Hi, > >> Sabri Berisha wrote on 01/08/2020 20:03: >>> but because Noction's decision to not enable NO_EXPORT by default >> >> the primary problem is not this b

Re: BGP route hijack by AS10990

On 1/Aug/20 21:31, Owen DeLong wrote: > I disagree. I think Noction and Telia are both culpable here. Most of the top > 200 providers > manage to do prefix filtering at the customer edge, so I don’t see any reason > to give > Telia a free pass here. Both Noction and Telia are culpable, becau

Re: BGP route hijack by AS10990

On 1/Aug/20 21:20, Owen DeLong wrote: > IP Prefix level filtering at the customer edge is not that hard, no > matter how large of a transit > provider you are. Customer edge filtration by Telia in this case would > have prevented this > problem from spreading beyond the misconfigured ASN. +1.

Re: BGP route hijack by AS10990

On 1/Aug/20 21:03, Sabri Berisha wrote: > The same can be said here. Noction and/or its operators appear to not > understand > how BGP works, and/or what safety measures must be deployed to ensure that the > larger internet will not be hurt by misconfiguration. I think the latter would be mor

Re: BGP route hijack by AS10990

On 1/Aug/20 20:14, Hank Nussbacher wrote: > AS  level filtering is easy.  IP prefix level filtering is hard.  > Especially when you are in the top 200: > > https://asrank.caida.org/ > Doesn't immediately make sense to me why prefix filtering is hard. > > That being said, and due to these BGP

Re: BGP route hijack by AS10990

Sabri Berisha wrote on 01/08/2020 20:59: My point is that there can be operational reasons to do so, and whatever they wish to do on their network is perfectly fine. As long as they don't bother the rest of the world with it. I get what you're saying, and am a big fan of personal responsibility

Re: BGP route hijack by AS10990

- On Aug 1, 2020, at 12:50 PM, Nick Hilliard n...@foobar.org wrote: Hi, > Sabri Berisha wrote on 01/08/2020 20:03: >> but because Noction's decision to not enable NO_EXPORT by default > > the primary problem is not this but that Noction reinjects prefixes into > the local ibgp mesh with the

Re: BGP route hijack by AS10990

Sabri Berisha wrote on 01/08/2020 20:03: but because Noction's decision to not enable NO_EXPORT by default the primary problem is not this but that Noction reinjects prefixes into the local ibgp mesh with the as-path stripped and then prioritises these prefixes so that they're learned as the

Re: BGP route hijack by AS10990

> On Aug 1, 2020, at 12:03 , Sabri Berisha wrote: > > Hi, > > - On Aug 1, 2020, at 8:49 AM, Owen DeLong o...@delong.com wrote: > >> In fact, there are striking parallels between Asiana 214 and this incident. > > Yes. Children of the magenta line. Depending on automation, and no clue wha

Re: BGP route hijack by AS10990

> On Aug 1, 2020, at 11:14 , Hank Nussbacher wrote: > > On 01/08/2020 00:50, Mark Tinka wrote: >> On 31/Jul/20 23:38, Sabri Berisha wrote: >> >>> Kudos to Telia for admitting their mistakes, and fixing their processes. >> Considering Telia's scope and "experience", that is one thing. But for >

Re: BGP route hijack by AS10990

Hi, - On Aug 1, 2020, at 8:49 AM, Owen DeLong o...@delong.com wrote: > In fact, there are striking parallels between Asiana 214 and this incident. Yes. Children of the magenta line. Depending on automation, and no clue what to do when the Instrument Landing System goes down. But, the most i

Re: BGP route hijack by AS10990

On 1/Aug/20 18:46, Owen DeLong wrote: > ROFLMAO, if you truly believe this, you have no concept of life in the > cockpit. I was born into aviation, with both my mom and dad licensed ATPL pilots for several decades. So I know my way around a number of different cockpits. The goal wasn't to tur

Re: BGP route hijack by AS10990

On 01/08/2020 00:50, Mark Tinka wrote: On 31/Jul/20 23:38, Sabri Berisha wrote: Kudos to Telia for admitting their mistakes, and fixing their processes. Considering Telia's scope and "experience", that is one thing. But for the general go

Re: BGP route hijack by AS10990

> On Aug 1, 2020, at 09:09 , Mark Tinka wrote: > > > > On 1/Aug/20 17:49, Owen DeLong wrote: > >> Aviation makes a strong effort in this area, perhaps stronger than any other >> human endeavor, especially when you’re talking about the fraction of >> Aviation known in the US as “Part 121 Sch

Re: BGP route hijack by AS10990

On 1/Aug/20 17:49, Owen DeLong wrote: > Aviation makes a strong effort in this area, perhaps stronger than any other > human endeavor, especially when you’re talking about the fraction of > Aviation known in the US as “Part 121 Scheduled Air Carrier Services”. > > However, as noted above, there

Re: BGP route hijack by AS10990

> On Aug 1, 2020, at 04:20 , Mark Tinka wrote: > > > > On 1/Aug/20 02:17, Sabri Berisha wrote: > >> I'm not sure if you read their entire Mea Culpa, but they did indicate that >> the root cause of this issue was the provisioning of a legacy filter that >> they are no longer using. So effect

Re: BGP route hijack by AS10990

On 1/Aug/20 16:44, Nick Hilliard wrote: > ... so once again, route optimisers were at the heart of another > serious route leaking incident. > > BGP is designed to prevent loops from happening, and has tools like > no-export to help prevent inadvertent leaks. > > When people build "BGP optimise

Re: BGP route hijack by AS10990

On 1/Aug/20 15:50, Ca By wrote: > > Aviation is regulated. Which is my point. While, like you, I am not in support in heavy-handed regulation like most life & death industries require, we also can't be leaving our industry open for any actor to do as they please. > > I am not normally support

Re: BGP route hijack by AS10990

Mark Tinka wrote on 01/08/2020 12:20: The difference between us and aviation is that fundamental flaws or mistakes that impact safety are required to be fixed and checked if you want to keep operating in the industry. We don't have that, so... ... so once again, route optimisers were at the hea

Re: BGP route hijack by AS10990

On Sat, Aug 1, 2020 at 4:21 AM Mark Tinka wrote: > > > What I meant by "TOTALLY avoidable" is that "this particular plane > crash" has happened in the exact same way, for the exact same reasons, > over and over again. > > Aviation learns from mistakes that don't generally recur in the exact > sam

Re: BGP route hijack by AS10990

On 1/Aug/20 02:17, Sabri Berisha wrote: > I'm not sure if you read their entire Mea Culpa, but they did indicate that > the root cause of this issue was the provisioning of a legacy filter that > they are no longer using. So effectively, that makes it a human error. > > We're going to a point w

Re: BGP route hijack by AS10990

- On Jul 31, 2020, at 2:50 PM, Mark Tinka mark.ti...@seacom.com wrote: Hi Mark, > On 31/Jul/20 23:38, Sabri Berisha wrote: > >> Kudos to Telia for admitting their mistakes, and fixing their processes. > > It's great that they are fixing this - but this was TOTALLY avoidable. I'm not sure i

Re: BGP route hijack by AS10990

On 31/Jul/20 23:38, Sabri Berisha wrote: > Kudos to Telia for admitting their mistakes, and fixing their processes. Considering Telia's scope and "experience", that is one thing. But for the general good of the Internet, the number of intended or unintentional route hijacks in recent years, an

Re: BGP route hijack by AS10990

- On Jul 31, 2020, at 2:33 PM, Lukas Tribus li...@ltri.eu wrote: Hi, > Telia's statement: > > https://blog.teliacarrier.com/2020/07/31/bgp-hijack-of-july-30-2020/ > > (tl;dr: it was as-path filtering only, as opposed to prefix filtering, > the former has been removed as an option) Kudos to

Re: BGP route hijack by AS10990

Telia's statement: https://blog.teliacarrier.com/2020/07/31/bgp-hijack-of-july-30-2020/ (tl;dr: it was as-path filtering only, as opposed to prefix filtering, the former has been removed as an option)

Re: BGP route hijack by AS10990

On 31/Jul/20 16:29, Mike Hammett wrote: > They solve a need that isn't reasonably solved any other way that > doesn't have similar drawbacks. > > Some optimizers need to be redesigned to be safer by default. > > Some networks need to be safer by default as well. Almost every product ever made do

Re: BGP route hijack by AS10990

On 31/Jul/20 16:07, Job Snijders wrote: > Could it be ... we didn't see any RPKI Invalids through Telia *because* > they are rejecting RPKI invalids? > > As far as I know the BGP Polluter software does not have a configuration > setting to only ruin the day of operators without ROAs. :-) > > I

Re: BGP route hijack by AS10990

On 31/Jul/20 16:01, Baldur Norddahl wrote: > How do you know that none of the prefixes had ROA? The ones that had > got stopped by Telias filter, so we would never know. Like I said, "if". If they did, then they were protected. If they didn't, well... > > This is exactly the situation where R

Re: BGP route hijack by AS10990

ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Mark Tinka" To: nanog@nanog.org Sent: Friday, July 31, 2020 8:59:51 AM Subject: Re: BGP route hijack by AS10990 On 30/Jul/20 19:44, Tom Beecher wrote: > It's not like there are scor

Re: BGP route hijack by AS10990

> > So while I will continue pushing for the rest of the world to create > ROA's, turn on RPKI and enable ROV, I'll also advocate that operators > continue to have both AS- and prefix-based filters. Not either/or, but > both. Also, max-prefix as a matter of course. > This is the correct approach.

Re: BGP route hijack by AS10990

On Fri, Jul 31, 2020 at 03:34:47PM +0200, Mark Tinka wrote: > On 31/Jul/20 03:57, Aftab Siddiqui wrote: > > Not a single prefix was signed, what I saw. May be good reason for > > Rogers, Charter, TWC etc to do that now. It would have stopped the > > propagation at Telia. > > If none of the prefixes

Re: BGP route hijack by AS10990

How do you know that none of the prefixes had ROA? The ones that had got stopped by Telias filter, so we would never know. This is exactly the situation where RPKI already works. My and yours prefixes, provided you like me have ROAs, will not be leaked through Telia and a number of other large tra

Re: BGP route hijack by AS10990

On 30/Jul/20 19:44, Tom Beecher wrote: > It's not like there are scorecards, but there's a lot of fault to go > around.  > > However, again, BGP "Optimizers" are bad. The conditions by which the > inadvertent leak occur need to be fixed , no question. But in > scenarios like this, as-path length

Re: BGP route hijack by AS10990

On 31/Jul/20 03:57, Aftab Siddiqui wrote: > Not a single prefix was signed, what I saw. May be good reason for > Rogers, Charter, TWC etc to do that now. It would have stopped the > propagation at Telia. While I am a huge proponent for ROA's and ROV, it is a massive expectation to req filtering

Re: BGP route hijack by AS10990

On 31/Jul/20 10:47, Nick Hilliard wrote:   > > Misconfig or oversight? We started using Telia as an upstream back in 2014. When we had new prefixes to announce to the Internet, we always sent them (as we do to all our upstreams) a request to update their filters to support the same. The standar

Re: BGP route hijack by AS10990

On 31.07.2020 10.47, Nick Hilliard wrote: Hank Nussbacher wrote on 31/07/2020 08:21: But wait - MANRS indicates that Telia does everything right: Not only that, Telia indicates that Telia does everything right: https://www.teliacarrier.com/our-network/bgp-routing/routing-security-.html

Re: BGP route hijack by AS10990

Hank Nussbacher wrote on 31/07/2020 08:21: But wait - MANRS indicates that Telia does everything right: Not only that, Telia indicates that Telia does everything right: https://www.teliacarrier.com/our-network/bgp-routing/routing-security-.html "We reject RPKI Invalids on all BGP Sessions;

Re: BGP route hijack by AS10990

On 30/07/2020 20:32, Sadiq Saif wrote: On Thu, 30 Jul 2020, at 13:09, Patrick Schultz wrote: so, bgp optimizers... again? -- Patrick More like shame on Telia for not filtering properly. But wait - MANRS indicates that Telia doe

Re: BGP route hijack by AS10990

Not a single prefix was signed, what I saw. May be good reason for Rogers, Charter, TWC etc to do that now. It would have stopped the propagation at Telia. On Fri, 31 Jul 2020 at 8:40 am, Baldur Norddahl wrote: > Telia implements RPKI filtering so the question is did it work? Were any > affected

Re: BGP route hijack by AS10990

Telia implements RPKI filtering so the question is did it work? Were any affected prefixes RPKI signed? Would any prefixes have avoided being hijacked if RPKI signing had been in place? Regards Baldur - who had to turn off RPKI filtering at the request of JTAC to stop our mx204s from crashing :-(

Re: BGP route hijack by AS10990

I'd like to direct you to Job's writeup on this :) https://mailman.nanog.org/pipermail/nanog/2017-August/191897.html While these "optimizers" CAN be beneficial to the individual operator, they're apparently used incorrectly in some instances. Telia should've filtered, that's for sure. But the lea

Re: BGP route hijack by AS10990

On Thu, Jul 30, 2020 at 07:09:07PM +0200, Patrick Schultz wrote: > so, bgp optimizers... again? We should stop calling them 'optimizers'... perhaps "BGP Polluters"? Kind regards, Job

Re: BGP route hijack by AS10990

> On Jul 30, 2020, at 09:45 , Yang Yu wrote: > > On Thu, Jul 30, 2020 at 9:37 AM Owen DeLong wrote: >> >> Looks like the real question here is why doesn’t 7219 do a better job of >> filtering what they accept. >> >> Has anyone reached out to them? > > You mean 1299? 7219 and 10990 are the

Re: BGP route hijack by AS10990

It's not like there are scorecards, but there's a lot of fault to go around. However, again, BGP "Optimizers" are bad. The conditions by which the inadvertent leak occur need to be fixed , no question. But in scenarios like this, as-path length generally limits impact to "Oh crap, I'll fix that, s

Re: BGP route hijack by AS10990

Peace, On Thu, Jul 30, 2020, 8:09 PM Patrick Schultz wrote: > so, bgp optimizers... again? > Looks so. Upstream filters are also to blame, though, but BGP optimization is the root of all evil. -- Töma >

Re: BGP route hijack by AS10990

On Thu, 30 Jul 2020, at 13:09, Patrick Schultz wrote: > so, bgp optimizers... again? > > -- > Patrick More like shame on Telia for not filtering properly. If Tulix used a so called BGP "optimizer" and didn't have a proper export filter in place it is their mistake but as a major transit provid

Re: BGP route hijack by AS10990

so, bgp optimizers... again? -- Patrick Am 30.07.2020 um 18:58 schrieb Töma Gavrichenkov: > Peace, > > On Thu, Jul 30, 2020, 5:48 AM Clinton Work > wrote: > > We saw a bunch of our IP blocks hijacked by AS10990 from 19:15 MDT until > 20:23 MDT.   Anybody else ha

Re: BGP route hijack by AS10990

Peace, On Thu, Jul 30, 2020, 5:48 AM Clinton Work wrote: > We saw a bunch of our IP blocks hijacked by AS10990 from 19:15 MDT until > 20:23 MDT. Anybody else have problems with that. > Here's what we discovered about the incident. Hope that brings some clarity. https://radar.qrator.net/blog

Re: BGP route hijack by AS10990

On Thu, Jul 30, 2020 at 9:37 AM Owen DeLong wrote: > > Looks like the real question here is why doesn’t 7219 do a better job of > filtering what they accept. > > Has anyone reached out to them? You mean 1299? 7219 and 10990 are the same entity.

Re: BGP route hijack by AS10990

Looks like the real question here is why doesn’t 7219 do a better job of filtering what they accept. Has anyone reached out to them? Owen > On Jul 29, 2020, at 23:31 , Aftab Siddiqui wrote: > > Looks like the list is too long.. none of them have any valid ROAs as well. > > = 104.230.0.0/18

Re: BGP route hijack by AS10990

On Thu, Jul 30, 2020 at 11:21:04AM +0300, Hank Nussbacher wrote a message of 48 lines which said: >See: And: https://stat.ripe.net/widget/bgp-update-activity#w.starttime=2020-07-16T05%3A00%3A00&w.endtime=2020-07-30T05%3A00%3A00&w.resource=AS10990

Re: BGP route hijack by AS10990

On 30/07/2020 05:46, Clinton Work wrote: See: https://bgpstream.com/event/245264 https://bgpstream.com/event/245265 -Hank Caveat: The views expressed above are solely my own and do not express the views or opinions of my

Re: BGP route hijack by AS10990

Looks like the list is too long.. none of them have any valid ROAs as well. = 104.230.0.0/18 206313 6724 1299 7219 10990 = 104.230.64.0/18 206313 6724 1299 7219 10990 = 107.184.0.0/16 206313 6724 1299 7219 10990 = 107.185.0.0/16 206313 6724 1299 7219 10990 = 107.189.192.0/19 206313 6724 1299 7219

Re: BGP route hijack by AS10990

We appeared to be impacted with some address space within 206.47.0.0/16 which AS577 normally advertises, but that was between 15:50 and 16:30 Eastern. Jeff On Wed, Jul 29, 2020, 10:48 PM Clinton Work wrote: > We saw a bunch of our IP blocks hijacked by AS10990 from 19:15 MDT until > 20:23 MDT.