Re: More specifics from AS18978 [was: Prefix hijack by INDOSAT AS4795 / AS4761]
Ignore my noise, I don't think there was new activity today (although something weird def. happened). BGPmon list was sorted by wrong column and I mixed the dates up. Although it's still showing as active since march which I thought said provider resolved... On 03/26/2015 8:26 pm, ML wrote: Wouldn't it be a BCP to set no-export from the Noction device too? On 3/26/2015 6:20 PM, Nick Rose wrote: Several people asked me off list for more details, here is what I have regarding it. This morning a tier2 isp that connects to our network made an error in their router configuration causing the route leakage. The issue has been addressed and we will be performing a full post mortem to ensure this does not happen again. While investigating the issue we did find that the noction appliance stopped advertising the no export community string with its advertisements which is why certain prefixes were also seen. Regards, Nick Rose CTO @ Enzu Inc. -Original Message- From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Nick Rose Sent: Thursday, March 26, 2015 3:49 PM To: a...@djlab.com; Peter Rocca Cc: nanog@nanog.org Subject: RE: More specifics from AS18978 [was: Prefix hijack by INDOSAT AS4795 / AS4761] This should be resolved from AS18978. If you experience anything else please let me know and I will get it addressed immediately. Regards, Nick Rose CTO @ Enzu Inc. -Original Message- From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Randy Sent: Thursday, March 26, 2015 12:14 PM To: Peter Rocca Cc: nanog@nanog.org Subject: RE: More specifics from AS18978 [was: Prefix hijack by INDOSAT AS4795 / AS4761] On 03/26/2015 9:00 am, Peter Rocca wrote: +1 The summary below aligns with our analysis as well. We've reached out to AS18978 to determine the status of the leak but at this time we're not seeing any operational impact. +2, after the morning coffee sunk in and helpful off list replies I can finally see it's probably not INDOSAT involved at all. FYI, the more specifics are still active: 2015-03-26 13:56:11 Update AS4795 ID 198.98.180.0/23 4795 4795 4761 9304 40633 18978 6939 29889 Active 2015-03-26 13:56:11 Update AS4795 ID 198.98.182.0/23 4795 4795 4761 9304 40633 18978 6939 29889 Active -- ~Randy
Re: More specifics from AS18978 [was: Prefix hijack by INDOSAT AS4795 / AS4761]
I guess AS18978 didn't learn from their mistake. Got a slew of identical bgpmon alerts for withdrawals and more specifics within the last 30 minutes. Worse than last time. Some still active, like: update time (UTC) Update Type Probe ASn Probe Location Prefix AS path Cleared Duration 2015-03-26 12:18:41 Update AS4795 ID 198.98.180.0/23 4795 4795 4761 9304 40633 18978 4436 29889 Active On 03/26/2015 8:26 pm, ML wrote: Wouldn't it be a BCP to set no-export from the Noction device too? On 3/26/2015 6:20 PM, Nick Rose wrote: Several people asked me off list for more details, here is what I have regarding it. This morning a tier2 isp that connects to our network made an error in their router configuration causing the route leakage. The issue has been addressed and we will be performing a full post mortem to ensure this does not happen again. While investigating the issue we did find that the noction appliance stopped advertising the no export community string with its advertisements which is why certain prefixes were also seen. Regards, Nick Rose CTO @ Enzu Inc. -Original Message- From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Nick Rose Sent: Thursday, March 26, 2015 3:49 PM To: a...@djlab.com; Peter Rocca Cc: nanog@nanog.org Subject: RE: More specifics from AS18978 [was: Prefix hijack by INDOSAT AS4795 / AS4761] This should be resolved from AS18978. If you experience anything else please let me know and I will get it addressed immediately. Regards, Nick Rose CTO @ Enzu Inc. -Original Message- From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Randy Sent: Thursday, March 26, 2015 12:14 PM To: Peter Rocca Cc: nanog@nanog.org Subject: RE: More specifics from AS18978 [was: Prefix hijack by INDOSAT AS4795 / AS4761] On 03/26/2015 9:00 am, Peter Rocca wrote: +1 The summary below aligns with our analysis as well. We've reached out to AS18978 to determine the status of the leak but at this time we're not seeing any operational impact. +2, after the morning coffee sunk in and helpful off list replies I can finally see it's probably not INDOSAT involved at all. FYI, the more specifics are still active: 2015-03-26 13:56:11 Update AS4795 ID 198.98.180.0/23 4795 4795 4761 9304 40633 18978 6939 29889 Active 2015-03-26 13:56:11 Update AS4795 ID 198.98.182.0/23 4795 4795 4761 9304 40633 18978 6939 29889 Active -- ~Randy
Re: More specifics from AS18978
On 27/Mar/15 12:03, Job Snijders wrote: Sure, but even that might not always prevent the fake paths from leaking to your eBGP neighbors. For instance, not too long ago there was this bug: "Routes learned with the no-export community from an iBGP neighbor are being advertised to eBGP neighbors. This may occur on Cisco ASR 9000 Series Aggregation Services Routers." (don't remember BugID) In other words: it can happen to the best of us. Your upstream could also re-write any BGP communities you attach to your BGP updates; so unless co-ordinated, there is no real guarantee a NO_EXPORT community will be maintained/honoured within your upstream's network. Mark.
Re: More specifics from AS18978
On Thu, Mar 26, 2015 at 11:26:07PM -0400, ML wrote: > On 3/26/2015 6:20 PM, Nick Rose wrote: > >While investigating the issue we did find that the noction appliance > >stopped advertising the no export community string with its > >advertisements which is why certain prefixes were also seen. > > Wouldn't it be a BCP to set no-export from the Noction device too? Sure, but even that might not always prevent the fake paths from leaking to your eBGP neighbors. For instance, not too long ago there was this bug: "Routes learned with the no-export community from an iBGP neighbor are being advertised to eBGP neighbors. This may occur on Cisco ASR 9000 Series Aggregation Services Routers." (don't remember BugID) In other words: it can happen to the best of us. You should not lie to yourself by inserting fake more-specific paths into routing tables. The moment your lies somehow manage to escape into the default-free-zone you are taking other businesses down. Whether the leak is caused by a bug in the router's software or human error, destroying other people's online presence is far beyond acceptable. If the same leak would've happened /without/ the fake more-specifics, it'd still be an issue, but the collateral damage would have been dampened. The leaked paths would have to compete with the normal paths and best-path selectors like as-path length apply. Using software to insert fake more-specific paths into your routing domain should be discouraged and frowned upon. Kind regards, Job
Re: More specifics from AS18978 [was: Prefix hijack by INDOSAT AS4795 / AS4761]
Wouldn't it be a BCP to set no-export from the Noction device too? On 3/26/2015 6:20 PM, Nick Rose wrote: Several people asked me off list for more details, here is what I have regarding it. This morning a tier2 isp that connects to our network made an error in their router configuration causing the route leakage. The issue has been addressed and we will be performing a full post mortem to ensure this does not happen again. While investigating the issue we did find that the noction appliance stopped advertising the no export community string with its advertisements which is why certain prefixes were also seen. Regards, Nick Rose CTO @ Enzu Inc. -Original Message- From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Nick Rose Sent: Thursday, March 26, 2015 3:49 PM To: a...@djlab.com; Peter Rocca Cc: nanog@nanog.org Subject: RE: More specifics from AS18978 [was: Prefix hijack by INDOSAT AS4795 / AS4761] This should be resolved from AS18978. If you experience anything else please let me know and I will get it addressed immediately. Regards, Nick Rose CTO @ Enzu Inc. -Original Message- From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Randy Sent: Thursday, March 26, 2015 12:14 PM To: Peter Rocca Cc: nanog@nanog.org Subject: RE: More specifics from AS18978 [was: Prefix hijack by INDOSAT AS4795 / AS4761] On 03/26/2015 9:00 am, Peter Rocca wrote: +1 The summary below aligns with our analysis as well. We've reached out to AS18978 to determine the status of the leak but at this time we're not seeing any operational impact. +2, after the morning coffee sunk in and helpful off list replies I can finally see it's probably not INDOSAT involved at all. FYI, the more specifics are still active: 2015-03-26 13:56:11 Update AS4795 ID 198.98.180.0/23 4795 4795 4761 9304 40633 18978 6939 29889 Active 2015-03-26 13:56:11 Update AS4795 ID 198.98.182.0/23 4795 4795 4761 9304 40633 18978 6939 29889 Active -- ~Randy
RE: More specifics from AS18978 [was: Prefix hijack by INDOSAT AS4795 / AS4761]
Several people asked me off list for more details, here is what I have regarding it. This morning a tier2 isp that connects to our network made an error in their router configuration causing the route leakage. The issue has been addressed and we will be performing a full post mortem to ensure this does not happen again. While investigating the issue we did find that the noction appliance stopped advertising the no export community string with its advertisements which is why certain prefixes were also seen. Regards, Nick Rose CTO @ Enzu Inc. -Original Message- From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Nick Rose Sent: Thursday, March 26, 2015 3:49 PM To: a...@djlab.com; Peter Rocca Cc: nanog@nanog.org Subject: RE: More specifics from AS18978 [was: Prefix hijack by INDOSAT AS4795 / AS4761] This should be resolved from AS18978. If you experience anything else please let me know and I will get it addressed immediately. Regards, Nick Rose CTO @ Enzu Inc. -Original Message- From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Randy Sent: Thursday, March 26, 2015 12:14 PM To: Peter Rocca Cc: nanog@nanog.org Subject: RE: More specifics from AS18978 [was: Prefix hijack by INDOSAT AS4795 / AS4761] On 03/26/2015 9:00 am, Peter Rocca wrote: > +1 > > The summary below aligns with our analysis as well. > > We've reached out to AS18978 to determine the status of the leak but > at this time we're not seeing any operational impact. +2, after the morning coffee sunk in and helpful off list replies I can finally see it's probably not INDOSAT involved at all. FYI, the more specifics are still active: 2015-03-26 13:56:11 Update AS4795 ID 198.98.180.0/23 4795 4795 4761 9304 40633 18978 6939 29889 Active 2015-03-26 13:56:11 Update AS4795 ID 198.98.182.0/23 4795 4795 4761 9304 40633 18978 6939 29889 Active -- ~Randy
RE: More specifics from AS18978 [was: Prefix hijack by INDOSAT AS4795 / AS4761]
This should be resolved from AS18978. If you experience anything else please let me know and I will get it addressed immediately. Regards, Nick Rose CTO @ Enzu Inc. -Original Message- From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Randy Sent: Thursday, March 26, 2015 12:14 PM To: Peter Rocca Cc: nanog@nanog.org Subject: RE: More specifics from AS18978 [was: Prefix hijack by INDOSAT AS4795 / AS4761] On 03/26/2015 9:00 am, Peter Rocca wrote: > +1 > > The summary below aligns with our analysis as well. > > We've reached out to AS18978 to determine the status of the leak but > at this time we're not seeing any operational impact. +2, after the morning coffee sunk in and helpful off list replies I can finally see it's probably not INDOSAT involved at all. FYI, the more specifics are still active: 2015-03-26 13:56:11 Update AS4795 ID 198.98.180.0/23 4795 4795 4761 9304 40633 18978 6939 29889 Active 2015-03-26 13:56:11 Update AS4795 ID 198.98.182.0/23 4795 4795 4761 9304 40633 18978 6939 29889 Active -- ~Randy
RE: More specifics from AS18978 [was: Prefix hijack by INDOSAT AS4795 / AS4761]
On 03/26/2015 9:00 am, Peter Rocca wrote: +1 The summary below aligns with our analysis as well. We've reached out to AS18978 to determine the status of the leak but at this time we're not seeing any operational impact. +2, after the morning coffee sunk in and helpful off list replies I can finally see it's probably not INDOSAT involved at all. FYI, the more specifics are still active: 2015-03-26 13:56:11 Update AS4795 ID 198.98.180.0/23 4795 4795 4761 9304 40633 18978 6939 29889 Active 2015-03-26 13:56:11 Update AS4795 ID 198.98.182.0/23 4795 4795 4761 9304 40633 18978 6939 29889 Active -- ~Randy
