Re: CenturyLink Fiber Latency Issues (Seattle, WA)

I second this, most best effort Broadband cpe equipment will choke with lots of concurrent connections On Tue, Nov 2, 2021, 8:25 PM P C wrote: > If this is connection count related only, It is most likely an issue with > the CPE (router), NAT table, or similar. > > On Tue, Nov 2, 2021 at 8:21 AM

Re: CenturyLink Fiber Latency Issues (Seattle, WA)

If this is connection count related only, It is most likely an issue with the CPE (router), NAT table, or similar. On Tue, Nov 2, 2021 at 8:21 AM Neel Chauhan wrote: > I tried that back in September, it didn't work. It doesn't happen on my > hop but the one after that. Even a second GPON connect

Re: CenturyLink Fiber Latency Issues (Seattle, WA)

Hi, I have taught of an (hackish) workaround for now. Enable my Tor relays, but at the same time switch my non-Tor traffic to Verizon "LTE Home". Then hope my neighbors have service calls with CenturyLink which forces them to fix the issue (a tech told me about "capacity issues"). Monitor the

Re: CenturyLink Fiber Latency Issues (Seattle, WA)

I tried that back in September, it didn't work. It doesn't happen on my hop but the one after that. Even a second GPON connection shows the issues if one is running the offending traffic. The issue occurs even if I'm using 50 Mbps out of my 940. It may be bufferbloat on CL's side but they keep

RE: CenturyLink Fiber Latency Issues (Seattle, WA)

Neel, Sounds like buffer bloat. Run a speed test, whatever is your maximum for your download and upload take 10% away from it, and setup traffic shaping in OPNsense (https://docs.opnsense.org/manual/shaping.html) with those values. If the issue goes away, then you're exceeding the buffer of Centu

Re: Centurylink Boise Networking Oddness

It is a tad bit unusual yes, but not surprising from Boise. Our connections have to go out to Seattle or other neighboring states before hitting any major IX. Well, that and basing issues solely on ICMP echos and trace routes can be tricky, given that they are low priority and require hitting

Re: Centurylink Boise Networking Oddness

100ms to twitch for continental USA seems a bit absurd! On Fri, Oct 9, 2020 at 10:56 AM Brielle wrote: > > Im on a CenturyLink fiber connection in Boise. What is the problem you > are seeing exactly? Traceroute doesn’t look odd really. > > > Sent from my iPhone > > On Oct 9, 2020, at 8:40 AM,

Re: Centurylink Boise Networking Oddness

Im on a CenturyLink fiber connection in Boise. What is the problem you are seeing exactly? Traceroute doesn’t look odd really. Sent from my iPhone > On Oct 9, 2020, at 8:40 AM, Allen Smith via NANOG wrote: > >  > > I apologize for the noise, this seems like the kind of thing where it >

Re: CenturyLink -> Lumen

Matt Harris|Infrastructure Lead Engineer 816-256-5446|Direct Looking for something? Helpdesk Portal|Email Support|Billing Portal We build and deliver end-to-end IT solutions. On Wed, Sep 16, 2020 at 9:31 AM Tom Hill wrote: > On 16/09/2020 11:18, Matt Hoppes wrote: > > Quantum Fiber? Sounds like

Re: CenturyLink -> Lumen

On 16/09/2020 11:18, Matt Hoppes wrote: > Quantum Fiber?  Sounds like a misbranding. I highly doubt they are using > Quantum technology.  Very prescient for when it becomes commercially possible though, eh? :) -- Tom

Re: CenturyLink -> Lumen

I think this is just someone trying to pull the stock price out of the dumps by branding themselves a “tech company”. There are still things from the TWTelecom days they haven’t finished integrating into the control panel, this should be fun watching them try to change the name at the same time

Re: CenturyLink -> Lumen

Quantum Fiber? Sounds like a misbranding. I highly doubt they are using Quantum technology. That’s how Lowe’s got in a lawsuit for selling 8” boards that were 7.6” long and similar. > On Sep 16, 2020, at 12:53 AM, R. Leigh Hennig wrote: > >  > https://www.fiercetelecom.com/telecom/century

Re: Centurylink having a bad morning?

On 5/Sep/20 18:46, Randy Bush via NANOG wrote: > fracking dmark illegal header hacking :( > > apologies No worries :-). Mark.

Re: Centurylink having a bad morning?

On 5/Sep/20 17:58, Randy Bush via NANOG wrote: > spoken like a tier two Even "Tier 1's" aren't present everywhere :-)... Although I'm not sure whether having presence in every city means you are a Tier-anything either... Mark.

Re: Centurylink having a bad morning?

*From: *"Mark Tinka" *To: *nanog@nanog.org *Sent: *Thursday, September 3, 2020 1:35:46 PM *Subject: *Re: Centurylink having a bad morning? On 31/Aug/20 17:57, Bryan Holloway wrote: > Not everyone will peer with you, notably, AS3356 (unle

Re: Centurylink having a bad morning?

> [ off list ] fracking dmark illegal header hacking :( apologies

Re: Centurylink having a bad morning?

[ off list ] >> Oh, yes! Let's not start another "what's a tier one" war! > > Oh no, let's :-). > > We get over here in Africa as well. Local operators either calling > themselves Tier 1, or being called a Tier 1. Nonsensical. > > Years back, our Marketing team asked me to comment on the use of

Re: Centurylink having a bad morning?

From: "Mark Tinka via NANOG" To: "Tomas Lynch" Cc: "NANOG" Sent: Saturday, September 5, 2020 5:26:21 AM Subject: Re: Centurylink having a bad morning? On 4/Sep/20 23:41, Tomas Lynch wrote: Oh, yes! Let's not start another "what's a ti

Re: Centurylink having a bad morning?

1:35:46 PM Subject: Re: Centurylink having a bad morning? On 31/Aug/20 17:57, Bryan Holloway wrote: > Not everyone will peer with you, notably, AS3356 (unless you're big > enough, which few can say.) I think Tomas meant more diverse peering, not peering with CL. Mark.

Re: Centurylink having a bad morning?

On 4/Sep/20 23:41, Tomas Lynch wrote: > > Oh, yes! Let's not start another "what's a tier one" war! Oh no, let's :-). We get over here in Africa as well. Local operators either calling themselves Tier 1, or being called a Tier 1. Nonsensical. Years back, our Marketing team asked me to comment

Re: Centurylink having a bad morning?

On Thu, Sep 3, 2020 at 2:37 PM Mark Tinka wrote: > > > On 31/Aug/20 17:57, Bryan Holloway wrote: > > > Not everyone will peer with you, notably, AS3356 (unless you're big > > enough, which few can say.) > > I think Tomas meant more diverse peering, not peering with CL. > Oh, yes! Let's not start

Re: Centurylink having a bad morning?

On 31/Aug/20 17:57, Bryan Holloway wrote: > Not everyone will peer with you, notably, AS3356 (unless you're big > enough, which few can say.) I think Tomas meant more diverse peering, not peering with CL. Mark.

Re: Centurylink having a bad morning?

On Wed, Sep 2, 2020 at 11:02, Warren Kumari wrote: >> well, what you REALLY need is one of these: >> https://www.cru-inc.com/products/wiebetech/hotplug_field_kit_product/ >> > > Yeah, no... actually, hell no! > > That setup scares me, and I'm surprised that it can be sold at all, > even with many

Re: Centurylink having a bad morning?

On 2/Sep/20 05:53, Alain Hebert wrote: > >     Beat installing a Cisco 12k solo with 2x4's to align the mounting > holes... I still have those in a rack somewhere, trapping air, if you want to test that :-)... Mark.

Re: Centurylink having a bad morning?

On 31/Aug/20 16:33, Tomas Lynch wrote: > Maybe we are idealizing these so-called tier-1 carriers and we, > tier-ns, should treat them as what they really are: another AS. Accept > that they are going to fail and do our best to mitigate the impact on > our own networks, i.e. more peering. Bingo!

Re: Centurylink having a bad morning?

- > > *De :* Shawn L via NANOG > > *Envoyé :* mercredi 2 septembre 2020 13:15 > > *À :* nanog > > *Objet :* Re: Centurylink having a bad morning? > > > > We once moved a 3u server 30 miles between data centers this way. Plug > > redu

Re: Centurylink having a bad morning?

That is what the 5G router is for... ons. 2. sep. 2020 19.47 skrev Michael Hallgren : > While conserving connectivity? 😂 > > > -- > *De :* Shawn L via NANOG > *Envoyé :* mercredi 2 septembre 2020 13:15 > *À :* nanog > *Objet :* Re: Centuryl

Re: Centurylink having a bad morning?

While conserving connectivity? 😂 De : Shawn L via NANOG Envoyé : mercredi 2 septembre 2020 13:15 À : nanog Objet : Re: Centurylink having a bad morning? We once moved a 3u server 30 miles between data centers this way.  Plug redundant psu into a ups and 2

Re: Centurylink having a bad morning?

On Wed, Sep 2, 2020 at 12:00 AM Christopher Morrow wrote: > > On Tue, Sep 1, 2020 at 11:53 PM Alain Hebert wrote: > > > > As a coincidence... I was *thinking* of moving a 90TB SAN (with > > mechanical's) to another rack that way... skateboard, long fibers and long > > power cords =D > > >

Re: Centurylink having a bad morning?

If the client pays me a shit ton of money to make sure the server won't turn off, and they pay for the hardware to make it happen. I;d think about it. It's a like a colo move on hardmode. Its extremely stupid, and I would advise not doing it. Hell even when I migrated e911 server, we had a 20 min

Re: Centurylink having a bad morning?

On 9/2/20 1:49 PM, Nick Hilliard wrote: Shawn L via NANOG wrote on 02/09/2020 12:15: We once moved a 3u server 30 miles between data centers this way. Plug redundant psu into a ups and 2 people carried it out and put them in a vehicle. hopefully none of these server moves that people have be

Re: Centurylink having a bad morning?

Shawn L via NANOG wrote on 02/09/2020 12:15: We once moved a 3u server 30 miles between data centers this way. Plug redundant psu into a ups and 2 people carried it out and put them in a vehicle. hopefully none of these server moves that people have been talking about involved spinning disks.

Re: Centurylink having a bad morning?

We once moved a 3u server 30 miles between data centers this way. Plug redundant psu into a ups and 2 people carried it out and put them in a vehicle. Sent from my iPhone > On Sep 1, 2020, at 11:58 PM, Christopher Morrow > wrote: > > On Tue, Sep 1, 2020 at 11:53 PM Alain Hebert wrote:

Re: Centurylink having a bad morning?

On Tue, Sep 1, 2020 at 11:53 PM Alain Hebert wrote: > > As a coincidence... I was *thinking* of moving a 90TB SAN (with > mechanical's) to another rack that way... skateboard, long fibers and long > power cords =D > well, what you REALLY need is one of these: https://www.cru-inc.com/pro

Re: Centurylink having a bad morning?

    As a coincidence...  I was *thinking* of moving a 90TB SAN (with mechanical's) to another rack that way...  skateboard, long fibers and long power cords =D     Beat installing a Cisco 12k solo with 2x4's to align the mounting holes... - Alain Hebertahe

Re: Centurylink having a bad morning?

ng like this can not happen again. The basic design of BGP is such >>> that this should not happen easily if at all. They did something unwise. >>> Did they make a route reflector based on a database or something? >>> >>> Regards, >>> >>> Bald

Re: Centurylink having a bad morning?

On Mon, Aug 31, 2020 at 11:28 PM Bjørn Mork wrote: > Well, many of us are paying for redundant power supplies or redundant > REs, even if that doesn't make any difference when the chassis is on > fire. I guess most people know that, and still buy those redundant > components. > I buy it so I ca

Re: Centurylink having a bad morning?

Eric Kuhnke writes: > There's a number of enterprise end user type customers of 3356 that have > on-premises server rooms/hosting for their stuff. And they spend a lot of > money every month for a 'redundant' metro ethernet circuit that takes > diverse fiber paths from their business park office

Re: Centurylink having a bad morning?

re they mention that a lot of hosting > companies only have one uplink to be quizzical and also the fact that he > goes pretty close to implying that its Centurylink’s customers fault for > not having multiple paths to Cloudflare that don’t touch Centurylink a bit > puzzling. It could have jus

Re: Centurylink having a bad morning?

udflare that don’t touch Centurylink a bit puzzling. >>> It could have just been poorly written. >>> >>> >>> >>> >>> >>> From: NANOG On Behalf Of >>> Tom Beecher >>> Sent: Monday, August 31, 2020 9:26 AM >&g

Re: Centurylink having a bad morning?

le paths to Cloudflare that don’t touch Centurylink a bit puzzling. It >> could have just been poorly written. >> >> >> >> >> >> From: NANOG On Behalf Of >> Tom Beecher >> Sent: Monday, August 31, 2020 9:26 AM >> To: Hank Nussbacher

Re: Centurylink having a bad morning?

t; close to implying that its Centurylink’s customers fault for not having >> multiple paths to Cloudflare that don’t touch Centurylink a bit puzzling. >> It could have just been poorly written. >> >> >> >> >> >> *From:* NANOG *On >> Behalf Of

Re: Centurylink having a bad morning?

t;> >> >> From: NANOG On Behalf Of >> Tom Beecher >> Sent: Monday, August 31, 2020 9:26 AM >> To: Hank Nussbacher >> Cc: NANOG >> Subject: Re: Centurylink having a bad morning? >> >> >> >> https://blog.cloudflare.com/analysis-o

Re: Centurylink having a bad morning?

> > *From:* NANOG *On Behalf > Of *Tom Beecher > *Sent:* Monday, August 31, 2020 9:26 AM > *To:* Hank Nussbacher > *Cc:* NANOG > *Subject:* Re: Centurylink having a bad morning? > > > > https://blog.cloudflare.com/analysis-of-todays-centurylink-level-3-outage/ > &

RE: Centurylink having a bad morning?

. It could have just been poorly written. From: NANOG On Behalf Of Tom Beecher Sent: Monday, August 31, 2020 9:26 AM To: Hank Nussbacher Cc: NANOG Subject: Re: Centurylink having a bad morning? https://blog.cloudflare.com/analysis-of-todays-centurylink-level-3-outage/ I definitely found Mr

Re: Centurylink having a bad morning?

ending them but I am sure it isn’t intentional. *From:* NANOG mailto:thenap@nanog.org>> *On Behalf Of *Baldur Norddahl *Sent:* Sunday, August 30, 2020 9:28 AM *To:* nanog@nanog.org <mailto:nanog@n

Re: Centurylink having a bad morning?

>>> But I disagree in that it would be impossible. They need to make a good >>>> report telling exactly what went wrong and how they changed the design, so >>>> something like this can not happen again. The basic design of BGP is such >>>> that this should no

Re: Centurylink having a bad morning?

hing unwise. >>> Did they make a route reflector based on a database or something? >>> >>> Regards, >>> >>> Baldur >>> >>> On Sun, Aug 30, 2020 at 5:13 PM Mike Bolitho >>> wrote: >>> >>>> Exactly. And aski

Re: Centurylink having a bad morning?

n Sun, Aug 30, 2020, 8:10 AM Drew Weaver mailto:drew.wea...@thenap.com>> wrote: I’m not defending them but I am sure it isn’t intentional. From: NANOG mailto:thenap@nanog.org>> On Behalf Of Baldur Norddahl Sent: Sunday, August 30, 2020 9:28 AM To: nanog@nanog.org<mailto:nan

Re: Centurylink having a bad morning?

a route reflector based on a database or something? >> >> Regards, >> >> Baldur >> >> On Sun, Aug 30, 2020 at 5:13 PM Mike Bolitho >> wrote: >> >>> Exactly. And asking that they somehow prove this won't happen again is >>> impo

Re: Centurylink having a bad morning?

isn’t intentional. From: NANOG mailto:thenap@nanog.org>> On Behalf Of Baldur Norddahl Sent: Sunday, August 30, 2020 9:28 AM To: nanog@nanog.org<mailto:nanog@nanog.org> Subject: Re: Centurylink having a bad morning? How is that acceptable behaviour? I shall remember never to make

Re: Centurylink having a bad morning?

: >> >>> I’m not defending them but I am sure it isn’t intentional. >>> >>> >>> >>> *From:* NANOG *On >>> Behalf Of *Baldur Norddahl >>> *Sent:* Sunday, August 30, 2020 9:28 AM >>> *To:* nanog@nanog.org

Re: Centurylink having a bad morning?

e it isn’t intentional.   From: NANOG thenap@nanog.org> On Behalf Of Baldur Norddahl Sent: Sunday, August 30, 2020 9:28 AM To: nanog@nanog.org Subject: Re:

Re: Centurylink having a bad morning?

On Sun, 30 Aug 2020 at 20:00, Baldur Norddahl wrote: > Not really the point. BGP is designed such that if I take down the link, the > prefixes MUST be withdrawn within reasonable time. The self healing aspect of > the internet entirely depends on this. Clearly they have some kind of system > t

Re: Centurylink having a bad morning?

un, Aug 30, 2020, 8:10 AM Drew Weaver wrote: >> >>> I’m not defending them but I am sure it isn’t intentional. >>> >>> >>> >>> *From:* NANOG *On >>> Behalf Of *Baldur Norddahl >>> *Sent:* Sunday, August 30, 2020 9:28 AM >>&

Re: Centurylink having a bad morning? [EXTERNAL]

On 30/08/2020 18:22, Joseph Jenkins wrote: Well at least it looks like the issue is starting to resolve  and stuff is coming back up. On Sun, Aug 30, 2020 at 8:21 AM Matt Hoppes

Re: Centurylink having a bad morning?

sure it isn’t intentional. >>> >>> >>> >>> From: NANOG On Behalf Of >>> Baldur Norddahl >>> Sent: Sunday, August 30, 2020 9:28 AM >>> To: nanog@nanog.org >>> Subject: Re: Centurylink having a bad morning? >>> &

Re: Centurylink having a bad morning?

, Aug 30, 2020, 8:10 AM Drew Weaver wrote: > >> I’m not defending them but I am sure it isn’t intentional. >> >> >> >> *From:* NANOG *On >> Behalf Of *Baldur Norddahl >> *Sent:* Sunday, August 30, 2020 9:28 AM >> *To:* nanog@nanog.org >> *Subject

Re: Centurylink having a bad morning?

Yep Regards Ray Ludendorff On Aug 30, 2020, at 16:16, Jason Kuehl wrote:  I'm over in MA in a CL building, it's very much still broken. I shut down the interfaces to CL and now just using Comcast. On Sun, Aug 30, 2020 at 9:20 AM Andy Brezinsky mailto:a...@mbrez.com>> wrote: Started about

Re: Centurylink having a bad morning?

On Sun, Aug 30, 2020 at 5:21 PM Chris Adams wrote: > Once upon a time, Baldur Norddahl said: > > How is that acceptable behaviour? I shall remember never to make a > contract > > with these guys until they can prove that they won't advertise my > prefixes > > after I pull them. Under any circums

Re: Centurylink having a bad morning?

I’m not defending them but I am sure it isn’t intentional. >> >> >> >> >> >> >> >> *From:* NANOG >> >> *On Behalf Of *Baldur Norddahl >> >> >> *Sent:* Sunday, August 30, 2020 9:28 AM >> >> >> *To:* nanog@na

Re: Centurylink having a bad morning?

I've been burning before. I'll wait at least an hour before turning my links back on. On Sun, Aug 30, 2020 at 11:31 AM Job Snijders wrote: > I believe from this moment forward things are converging back to normal. > > Kind regards, > > Job > -- Sincerely, Jason W Kuehl Cell 920-419-8983 jaso

Re: Centurylink having a bad morning? [EXTERNAL]

Well at least it looks like the issue is starting to resolve and stuff is coming back up. On Sun, Aug 30, 2020 at 8:21 AM Matt Hoppes < mattli...@rivervalleyinternet.net> wrote: > Is this what happens when your entire network is database driven? >

Re: Centurylink having a bad morning?

; *From:* NANOG *On Behalf > Of *Baldur Norddahl > *Sent:* Sunday, August 30, 2020 9:28 AM > *To:* nanog@nanog.org > *Subject:* Re: Centurylink having a bad morning? > > > > How is that acceptable behaviour? I shall remember never to make a > contract with these guys un

Re: Centurylink having a bad morning?

*Sent:* Sunday, August 30, 2020 9:28 AM > *To:* nanog@nanog.org > *Subject:* Re: Centurylink having a bad morning? > > > > How is that acceptable behaviour? I shall remember never to make a > contract with these guys until they can prove that they won't advertise my &

Re: Centurylink having a bad morning?

AS3356 is the Level3 internet. On Sun, Aug 30, 2020 at 8:09 AM Ian Bowers wrote: > AS3356 is the one I've seen all the chatter about this morning. > > On Sun, Aug 30, 2020 at 10:03 AM Robert Blayzor > wrote: > >> On 8/30/20 8:14 AM, Drew Weaver via NANOG wrote: >> > Woke up this morning to a bu

RE: Centurylink having a bad morning?

Weaver Sent: Sunday, August 30, 2020 10:07 AM To: 'Ben Russell' ; 'nanog@nanog.org' Subject: RE: Centurylink having a bad morning? Yikes, they really need to get their reflectors fixed so that we can dis-engage from them. -Original Message- From: NANOG On Behalf

Re: Centurylink having a bad morning?

I believe from this moment forward things are converging back to normal. Kind regards, Job

Re: Centurylink having a bad morning?

10:00 AM > *To:* Drew Weaver > *Cc:* R. Leigh Hennig ; nanog@nanog.org > *Subject:* Re: Centurylink having a bad morning? > > > > People are rebooting ghosting now. > > > > https://twitter.com/ir_kujoe/status/1300066569645707265 > > > > Seeing other

Re: Centurylink having a bad morning?

atti or Ashburn… > > > > > > > > From: Tomas Lynch > > Sent: Sunday, August 30, 2020 8:45 AM > > To: Mel Beckman > > Cc: Drew Weaver ; nanog@nanog.org > > Subject: Re: Centurylink having a bad morning? > > > > > > > > BGP sessio

Re: Centurylink having a bad morning?

Once upon a time, Baldur Norddahl said: > How is that acceptable behaviour? I shall remember never to make a contract > with these guys until they can prove that they won't advertise my prefixes > after I pull them. Under any circumstances. Umm, then I guess you won't sign a contract with anybody

Re: Centurylink having a bad morning? [EXTERNAL]

Is this what happens when your entire network is database driven?

Re: Centurylink having a bad morning?

atti or Ashburn… > > > > > > > > From: Tomas Lynch > > Sent: Sunday, August 30, 2020 8:45 AM > > To: Mel Beckman > > Cc: Drew Weaver ; nanog@nanog.org > > Subject: Re: Centurylink having a bad morning? > > > > > > > > BGP

RE: Centurylink having a bad morning?

I’m not defending them but I am sure it isn’t intentional. From: NANOG On Behalf Of Baldur Norddahl Sent: Sunday, August 30, 2020 9:28 AM To: nanog@nanog.org Subject: Re: Centurylink having a bad morning? How is that acceptable behaviour? I shall remember never to make a contract with these

Re: Centurylink having a bad morning?

AS3356 is the one I've seen all the chatter about this morning. On Sun, Aug 30, 2020 at 10:03 AM Robert Blayzor wrote: > On 8/30/20 8:14 AM, Drew Weaver via NANOG wrote: > > Woke up this morning to a bunch of reports of issues with connectivity > > had to shut down some Level3/CTL connections to

Re: Centurylink having a bad morning?

:14:27 AM To: David Hubbard Cc: nanog@nanog.org Subject: Re: Centurylink having a bad morning? They’re not reachable so who knows if they’re even working on it. Gonna go out on a limb here and assume that a lot of phones were ringing and people are in fact working on whatever it is. :) On Sun

Re: Centurylink having a bad morning?

Once upon a time, Robert Blayzor said: > Just to confirm we're seeing this on AS3356 and not AS209, correct? Correct - we had problems with our 3356 connection but not our 209 connection. -- Chris Adams

RE: Centurylink having a bad morning?

Yikes, they really need to get their reflectors fixed so that we can dis-engage from them. -Original Message- From: NANOG On Behalf Of Ben Russell Sent: Sunday, August 30, 2020 9:14 AM To: 'nanog@nanog.org' Subject: RE: Centurylink having a bad morning? One of my legacy L

Re: Centurylink having a bad morning?

I just brought one of my sessions back up to attempt to avoid the blackholing, should be full feed, getting all of 850 v4 routes and 106 v6. From: NANOG on behalf of Tomas Lynch Date: Sunday, August 30, 2020 at 9:41 AM To: Drew Weaver Cc: "nanog@nanog.org" Subject: Re: Centurylin

Re: Centurylink having a bad morning?

Now if you call into CL you get a message stating their technicians are working on an ip outage. On Sun, Aug 30, 2020 at 6:56 AM Chase Christian wrote: > Multiple BGP sessions with Level3 (DIA) started flapping at approx 03:00 > Pacific: > > Aug 30 03:05:13 rtr02 Rib: %BGP-3-NOTIFICATION: sent t

RE: Centurylink having a bad morning?

Something just now changed in this situation and now it seems to have gotten worse. From: Jason Kuehl Sent: Sunday, August 30, 2020 10:00 AM To: Drew Weaver Cc: R. Leigh Hennig ; nanog@nanog.org Subject: Re: Centurylink having a bad morning? People are rebooting ghosting now. [cid:image001

Re: Centurylink having a bad morning?

actual network? > > > > *From:* R. Leigh Hennig > *Sent:* Sunday, August 30, 2020 8:54 AM > *To:* Drew Weaver ; nanog@nanog.org > *Subject:* Re: Centurylink having a bad morning? > > > > Global impact with issues reported by Fastly, Cloudflare, OpenDNS. > > >

RE: Centurylink having a bad morning? [EXTERNAL]

Their route-servers just can’t keep up. They are not getting our community-string announcements at all From: NANOG On Behalf Of Drew Weaver Sent: Sunday, August 30, 2020 8:38 AM To: 'David Hubbard' ; 'nanog@nanog.org' Subject: RE: Centurylink having a bad morning? [E

Re: Centurylink having a bad morning?

; > > From: Tomas Lynch > Sent: Sunday, August 30, 2020 8:45 AM > To: Mel Beckman > Cc: Drew Weaver ; nanog@nanog.org > Subject: Re: Centurylink having a bad morning? > > > > BGP sessions randomly flapping or having routing issues in different cities > si

Re: Centurylink having a bad morning?

How is that acceptable behaviour? I shall remember never to make a contract with these guys until they can prove that they won't advertise my prefixes after I pull them. Under any circumstances. søn. 30. aug. 2020 15.14 skrev Joseph Jenkins : > Finally got through on their support line and spoke

RE: Centurylink having a bad morning?

I don’t think it is anywhere near back to normal but I can get to 2-3 more sites than I could when I got called in. From: NANOG On Behalf Of Andy Brezinsky Sent: Sunday, August 30, 2020 8:48 AM To: nanog@nanog.org Subject: Re: Centurylink having a bad morning? Started about 5:05am central

Re: Centurylink having a bad morning?

I'm over in MA in a CL building, it's very much still broken. I shut down the interfaces to CL and now just using Comcast. On Sun, Aug 30, 2020 at 9:20 AM Andy Brezinsky wrote: > Started about 5:05am central, started clearing up for me about 7:15am. > My route from ATT in Chicago is still going

Re: Centurylink having a bad morning?

> > They’re not reachable so who knows if they’re even working on it. > Gonna go out on a limb here and assume that a lot of phones were ringing and people are in fact working on whatever it is. :) On Sun, Aug 30, 2020 at 8:34 AM David Hubbard wrote: > Same. Also, as reported on outages list,

RE: Centurylink having a bad morning?

One of my legacy L3 peers is only advertising 498174 prefixes. Very strange. Ben Russell  -Original Message- From: NANOG On Behalf Of Drew Weaver Sent: Sunday, August 30, 2020 7:45 AM To: 'Mel Beckman' Cc: 'nanog@nanog.org' Subject: RE: Centurylink having a

Re: Centurylink having a bad morning?

On 8/30/20 8:14 AM, Drew Weaver via NANOG wrote: > Woke up this morning to a bunch of reports of issues with connectivity > had to shut down some Level3/CTL connections to get it to return to normal. Just to confirm we're seeing this on AS3356 and not AS209, correct? We have links to both and

Re: Centurylink having a bad morning?

Multiple BGP sessions with Level3 (DIA) started flapping at approx 03:00 Pacific: Aug 30 03:05:13 rtr02 Rib: %BGP-3-NOTIFICATION: sent to neighbor 4.35.X.Y (AS 3356) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Aug 30 03:05:13 rtr02 Rib: %BGP-5-ADJCHANGE: peer 4.35.X.Y (AS 3356) old state Es

Re: Centurylink having a bad morning?

We've been on hold for more than an hour trying to get an update. We see the same behavior where they continue to announce our blocks despite all the interfaces to them being hard down. Scott Helms On Sun, Aug 30, 2020 at 8:58 AM Jason Kuehl wrote: > > Well, When I tried calling I got a fast b

RE: Centurylink having a bad morning?

That site seems to be just for their cloud products, is there one of these for their actual network? From: R. Leigh Hennig Sent: Sunday, August 30, 2020 8:54 AM To: Drew Weaver ; nanog@nanog.org Subject: Re: Centurylink having a bad morning? Global impact with issues reported by Fastly

Re: Centurylink having a bad morning?

natti or Ashburn… > > > > *From:* Tomas Lynch > *Sent:* Sunday, August 30, 2020 8:45 AM > *To:* Mel Beckman > *Cc:* Drew Weaver ; nanog@nanog.org > *Subject:* Re: Centurylink having a bad morning? > > > > BGP sessions randomly flapping or having routing issues in diff

RE: Centurylink having a bad morning?

Saw the flapping in Cleveland but not in Cincinnatti or Ashburn… From: Tomas Lynch Sent: Sunday, August 30, 2020 8:45 AM To: Mel Beckman Cc: Drew Weaver ; nanog@nanog.org Subject: Re: Centurylink having a bad morning? BGP sessions randomly flapping or having routing issues in different cities

Re: Centurylink having a bad morning?

Looks like this is causing problems with Meraki Cloud connectivity since level3 is a transit provider for Meraki. From: NANOG on behalf of Mel Beckman Sent: Sunday, August 30, 2020, 08:47 To: Drew Weaver Cc: nanog@nanog.org Subject: Re: Centurylink having a

Re: Centurylink having a bad morning?

I am seeing some odd traffic deflections in the EU via 3356, but nothing in the US so far. Some scattershot oddball reports landing in our NOC, but nothing conclusive. On Sun, Aug 30, 2020 at 8:41 AM Mel Beckman wrote: > The CL portal loads for me, and I can log in, but it is slower than usual.

Re: Centurylink having a bad morning?

Started about 5:05am central, started clearing up for me about 7:15am.   My route from ATT in Chicago is still going through NYC to get back to Chicago but at least packet loss isn't 70-100% anymore. I also tried turning down sessions and still was seeing stale announcements on other LGs. O

Re: Centurylink having a bad morning?

Finally got through on their support line and spoke to level1. The only thing the tech could say was it was an issue with BGP route reflectors and it started about 3am(pacific). They were still trying to isolate the issue. I've tried failing over my circuits and no go, the traffic just dies as L3 w

RE: Centurylink having a bad morning?

Can't get to pretty much anything on the Internet from my Tmobile device... not really sure whats going on =) -Original Message- From: Mel Beckman Sent: Sunday, August 30, 2020 8:43 AM To: Drew Weaver Cc: nanog@nanog.org Subject: Re: Centurylink having a bad morning? One anomaly

  1   2   3   >