Re: [AFMUG] Protecting Upstream Connections
We had a major regional cogent outage last night. Sent from my iPhone > On Nov 15, 2022, at 6:26 PM, Steve Jones wrote: > > > Our upstreams upstream just started rejecting our routes on sunday. Nobody > knows why yet, they're investigating > >> On Tue, Nov 15, 2022, 2:09 PM Nate Burke wrote: >> Ran into a problem today where one of my upstreams (windstream) was >> still advertising my IP blocks out to the internet, but wasn't routing >> all the traffic to me. I was still taking in traffic, but, for example, >> Google couldn't get to me on IP Blocks advertized via windstream. Took >> a while to track down what was happening and then manually turn off the >> Windstream BGP neighbor, which luckily fixed the problem once the subnet >> advertisements finally timed out and traffic shifted to other providers, >> which took a good 5-10 minutes. >> >> Is there a way to protect against that? My BGP Session to Windstream >> was still up and running, but something broke farther up in the >> Windstream network. Is the only method in that scenario to Just monitor >> and react? >> >> -- >> AF mailing list >> AF@af.afmug.com >> http://af.afmug.com/mailman/listinfo/af_af.afmug.com > -- > AF mailing list > AF@af.afmug.com > http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com
Re: [AFMUG] Protecting Upstream Connections
Our upstreams upstream just started rejecting our routes on sunday. Nobody knows why yet, they're investigating On Tue, Nov 15, 2022, 2:09 PM Nate Burke wrote: > Ran into a problem today where one of my upstreams (windstream) was > still advertising my IP blocks out to the internet, but wasn't routing > all the traffic to me. I was still taking in traffic, but, for example, > Google couldn't get to me on IP Blocks advertized via windstream. Took > a while to track down what was happening and then manually turn off the > Windstream BGP neighbor, which luckily fixed the problem once the subnet > advertisements finally timed out and traffic shifted to other providers, > which took a good 5-10 minutes. > > Is there a way to protect against that? My BGP Session to Windstream > was still up and running, but something broke farther up in the > Windstream network. Is the only method in that scenario to Just monitor > and react? > > -- > AF mailing list > AF@af.afmug.com > http://af.afmug.com/mailman/listinfo/af_af.afmug.com > -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com
Re: [AFMUG] Protecting Upstream Connections
Different boxes use different mechanisms to determine availability. One of them pings something like your top 10,000 destinations via each upstream every second, so then it knows what's up. - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "Nate Burke" To: "AnimalFarm Microwave Users Group" Sent: Tuesday, November 15, 2022 4:00:31 PM Subject: Re: [AFMUG] Protecting Upstream Connections Wouldn't there have to be something on the other end to determine if traffic can get to you? I was still taking some traffic, just Google, and I'm guessing some other CDN traffic couldn't get to me. Which to Eyeball networks means 'the internet is down' On 11/15/2022 3:44 PM, Mike Hammett wrote: You can have it pull the BGP advertisement or set no export or prepend a bunch or other things ranging from suggestion to absolute. In this case, if the optimizer noticed that you weren't available from a given connection, it could have (depending on the optimizer), set no export, then if the reachability issues persist after X minutes, drop the session. - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "TJ Trout" To: "AnimalFarm Microwave Users Group" Sent: Tuesday, November 15, 2022 3:21:40 PM Subject: Re: [AFMUG] Protecting Upstream Connections Is there any benefit for an eyeball network to use route optimization given you can really only control egress via bgp? On Tue, Nov 15, 2022, 12:26 PM Nate Burke < n...@blastcomm.com > wrote: I rarely have issues with the Windstream circuit, but it is totally dealing with a Telco. Front line can only open the ticket, and go through the whole list of questions that have been in place since troubleshooting a T1 Line, or before. I always like 'Are we authorized to do intrusive testing' This is only the 2nd time this has happened. Last time was 4 or 5 years ago, just a week or 2 after we turned up the circuit, so I wrote that off to a turnup config they messed up somewhere. On 11/15/2022 2:16 PM, Larry Smith wrote: > In the past when we had similar problems with Windstream I found > that often (about 75 percent of the time) I could hard reset the BGP > session locally and Windstream would pickup after the reset and > route correctly. The other 25 percent, like you, I just had to down > the BGP session with them entirely and wait for the routes to clear. > A few of those times I even called our 24/hr support number to ask > if they could clear or reset the session from their side and was > inevitably told "we will look into it" (not)... > -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com
Re: [AFMUG] Protecting Upstream Connections
Wouldn't there have to be something on the other end to determine if traffic can get to you? I was still taking some traffic, just Google, and I'm guessing some other CDN traffic couldn't get to me. Which to Eyeball networks means 'the internet is down' On 11/15/2022 3:44 PM, Mike Hammett wrote: You can have it pull the BGP advertisement or set no export or prepend a bunch or other things ranging from suggestion to absolute. In this case, if the optimizer noticed that you weren't available from a given connection, it could have (depending on the optimizer), set no export, then if the reachability issues persist after X minutes, drop the session. - Mike Hammett Intelligent Computing Solutions <http://www.ics-il.com/> <https://www.facebook.com/ICSIL><https://plus.google.com/+IntelligentComputingSolutionsDeKalb><https://www.linkedin.com/company/intelligent-computing-solutions><https://twitter.com/ICSIL> Midwest Internet Exchange <http://www.midwest-ix.com/> <https://www.facebook.com/mdwestix><https://www.linkedin.com/company/midwest-internet-exchange><https://twitter.com/mdwestix> The Brothers WISP <http://www.thebrotherswisp.com/> <https://www.facebook.com/thebrotherswisp> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg> *From: *"TJ Trout" *To: *"AnimalFarm Microwave Users Group" *Sent: *Tuesday, November 15, 2022 3:21:40 PM *Subject: *Re: [AFMUG] Protecting Upstream Connections Is there any benefit for an eyeball network to use route optimization given you can really only control egress via bgp? On Tue, Nov 15, 2022, 12:26 PM Nate Burke <mailto:n...@blastcomm.com>> wrote: I rarely have issues with the Windstream circuit, but it is totally dealing with a Telco. Front line can only open the ticket, and go through the whole list of questions that have been in place since troubleshooting a T1 Line, or before. I always like 'Are we authorized to do intrusive testing' This is only the 2nd time this has happened. Last time was 4 or 5 years ago, just a week or 2 after we turned up the circuit, so I wrote that off to a turnup config they messed up somewhere. On 11/15/2022 2:16 PM, Larry Smith wrote: > In the past when we had similar problems with Windstream I found > that often (about 75 percent of the time) I could hard reset the BGP > session locally and Windstream would pickup after the reset and > route correctly. The other 25 percent, like you, I just had to down > the BGP session with them entirely and wait for the routes to clear. > A few of those times I even called our 24/hr support number to ask > if they could clear or reset the session from their side and was > inevitably told "we will look into it" (not)... > -- AF mailing list AF@af.afmug.com <mailto:AF@af.afmug.com> http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com
Re: [AFMUG] Protecting Upstream Connections
You can have it pull the BGP advertisement or set no export or prepend a bunch or other things ranging from suggestion to absolute. In this case, if the optimizer noticed that you weren't available from a given connection, it could have (depending on the optimizer), set no export, then if the reachability issues persist after X minutes, drop the session. - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "TJ Trout" To: "AnimalFarm Microwave Users Group" Sent: Tuesday, November 15, 2022 3:21:40 PM Subject: Re: [AFMUG] Protecting Upstream Connections Is there any benefit for an eyeball network to use route optimization given you can really only control egress via bgp? On Tue, Nov 15, 2022, 12:26 PM Nate Burke < n...@blastcomm.com > wrote: I rarely have issues with the Windstream circuit, but it is totally dealing with a Telco. Front line can only open the ticket, and go through the whole list of questions that have been in place since troubleshooting a T1 Line, or before. I always like 'Are we authorized to do intrusive testing' This is only the 2nd time this has happened. Last time was 4 or 5 years ago, just a week or 2 after we turned up the circuit, so I wrote that off to a turnup config they messed up somewhere. On 11/15/2022 2:16 PM, Larry Smith wrote: > In the past when we had similar problems with Windstream I found > that often (about 75 percent of the time) I could hard reset the BGP > session locally and Windstream would pickup after the reset and > route correctly. The other 25 percent, like you, I just had to down > the BGP session with them entirely and wait for the routes to clear. > A few of those times I even called our 24/hr support number to ask > if they could clear or reset the session from their side and was > inevitably told "we will look into it" (not)... > -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com
Re: [AFMUG] Protecting Upstream Connections
Is there any benefit for an eyeball network to use route optimization given you can really only control egress via bgp? On Tue, Nov 15, 2022, 12:26 PM Nate Burke wrote: > I rarely have issues with the Windstream circuit, but it is totally > dealing with a Telco. Front line can only open the ticket, and go > through the whole list of questions that have been in place since > troubleshooting a T1 Line, or before. I always like 'Are we authorized > to do intrusive testing' > > This is only the 2nd time this has happened. Last time was 4 or 5 years > ago, just a week or 2 after we turned up the circuit, so I wrote that > off to a turnup config they messed up somewhere. > > On 11/15/2022 2:16 PM, Larry Smith wrote: > > In the past when we had similar problems with Windstream I found > > that often (about 75 percent of the time) I could hard reset the BGP > > session locally and Windstream would pickup after the reset and > > route correctly. The other 25 percent, like you, I just had to down > > the BGP session with them entirely and wait for the routes to clear. > > A few of those times I even called our 24/hr support number to ask > > if they could clear or reset the session from their side and was > > inevitably told "we will look into it" (not)... > > > > > -- > AF mailing list > AF@af.afmug.com > http://af.afmug.com/mailman/listinfo/af_af.afmug.com > -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com
Re: [AFMUG] Protecting Upstream Connections
I rarely have issues with the Windstream circuit, but it is totally dealing with a Telco. Front line can only open the ticket, and go through the whole list of questions that have been in place since troubleshooting a T1 Line, or before. I always like 'Are we authorized to do intrusive testing' This is only the 2nd time this has happened. Last time was 4 or 5 years ago, just a week or 2 after we turned up the circuit, so I wrote that off to a turnup config they messed up somewhere. On 11/15/2022 2:16 PM, Larry Smith wrote: In the past when we had similar problems with Windstream I found that often (about 75 percent of the time) I could hard reset the BGP session locally and Windstream would pickup after the reset and route correctly. The other 25 percent, like you, I just had to down the BGP session with them entirely and wait for the routes to clear. A few of those times I even called our 24/hr support number to ask if they could clear or reset the session from their side and was inevitably told "we will look into it" (not)... -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com
Re: [AFMUG] Protecting Upstream Connections
In the past when we had similar problems with Windstream I found that often (about 75 percent of the time) I could hard reset the BGP session locally and Windstream would pickup after the reset and route correctly. The other 25 percent, like you, I just had to down the BGP session with them entirely and wait for the routes to clear. A few of those times I even called our 24/hr support number to ask if they could clear or reset the session from their side and was inevitably told "we will look into it" (not)... -- Larry Smith lesm...@ecsis.net On Tue November 15 2022 14:08, Nate Burke wrote: > Ran into a problem today where one of my upstreams (windstream) was > still advertising my IP blocks out to the internet, but wasn't routing > all the traffic to me. I was still taking in traffic, but, for example, > Google couldn't get to me on IP Blocks advertized via windstream. Took > a while to track down what was happening and then manually turn off the > Windstream BGP neighbor, which luckily fixed the problem once the subnet > advertisements finally timed out and traffic shifted to other providers, > which took a good 5-10 minutes. > > Is there a way to protect against that? My BGP Session to Windstream > was still up and running, but something broke farther up in the > Windstream network. Is the only method in that scenario to Just monitor > and react? -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com
Re: [AFMUG] Protecting Upstream Connections
Monitor and react. That could be automated or manual. The BGP route optimizers could shut off that peer when they detect issues. - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "Nate Burke" To: "Animal Farm" Sent: Tuesday, November 15, 2022 2:08:51 PM Subject: [AFMUG] Protecting Upstream Connections Ran into a problem today where one of my upstreams (windstream) was still advertising my IP blocks out to the internet, but wasn't routing all the traffic to me. I was still taking in traffic, but, for example, Google couldn't get to me on IP Blocks advertized via windstream. Took a while to track down what was happening and then manually turn off the Windstream BGP neighbor, which luckily fixed the problem once the subnet advertisements finally timed out and traffic shifted to other providers, which took a good 5-10 minutes. Is there a way to protect against that? My BGP Session to Windstream was still up and running, but something broke farther up in the Windstream network. Is the only method in that scenario to Just monitor and react? -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com -- AF mailing list AF@af.afmug.com http://af.afmug.com/mailman/listinfo/af_af.afmug.com
