Re: [outages] CANNOT COMMENT BEYOND THIS : Two separate fiber cuts on major routes in US for CTL

This doesn’t come as a surprise to me for CenturyLink. We’ve had their IP services in Tampa taken down by simultaneous cuts between both Tampa and Miami, and Tampa and Dallas(!!). So, they’re clearly comfortable having no tolerance for more than one fiber cut across 1200+ miles of fiber. From:

Re: [outages] CANNOT COMMENT BEYOND THIS : Two separate fiber cuts on major routes in US for CTL

Heres the bad news now... Going to take awhile... *** CASCADED EXTERNAL NOTES 21-Apr-2020 18:58:04 GMT From CASE: 18573666 - Event IP NOC reports that at the first fiber cut location, the anticipated slack in the fiber span located in the secondary manhole for the south end of the span is una

[outages] CANNOT COMMENT BEYOND THIS : Two separate fiber cuts on major routes in US for CTL

Super bad timing. A crap situation and terrible timing. That's the 3ft view. The person that sent the public CTL notes has the right of it. Keep an eye out for similar from others. == Noah Weis Internet Network Architecture, CenturyLink e: noah.w...@centurylink.com

Re: [outages] Centurylink Packet Loss - Socal

Latest update... 04/21/2020 16:58:13 GMT - IP NOC reports teams at the first cut site continue working to pull slack which is being slowed by having to pump out a manhole. Prep work is in progress on one side of the cable. Teams at the second cut site have identified the damage caused by constr

Re: [outages] US West Coast Massive Tier 1 Backbone Issues

Great read on carrier diversity: https://transition.fcc.gov/bureaus/pshs/docs/clearinghouse/ATIS_NDAI_Final_Report_2006.pdf From: Outages [mailto:outages-boun...@outages.org] On Behalf Of Jay Ashworth via Outages Sent: Tuesday, April 21, 2020 9:50 AM To: Brett Dikeman Cc: Cary Wiedemann via Out

Re: [outages] Centurylink Packet Loss - Socal

Do you see packet loss on the point to point links or transit peers? On Tue, Apr 21, 2020 at 9:46 AM Erik Lentz via Outages wrote: > The occasional batch of dropped packets in Arkansas, old L3 network. > > On Tue, Apr 21, 2020 at 11:22 AM David Hubbard via Outages < > outages@outages.org> wrote:

Re: [outages] US West Coast Massive Tier 1 Backbone Issues

Try the 60,000 foot view; the midwest is having a bad day as well, though it sounds like a fiber cut is to blame. https://www.reddit.com/r/sysadmin/comments/g5gyms/midwest_internet_outage/ On Tue, Apr 21, 2020 at 12:38 PM Jay Ashworth via Outages < outages@outages.org> wrote: > I was hoping so

Re: [outages] US West Coast Massive Tier 1 Backbone Issues

Yeah, the CenturyLink ticket is running in another thread on the list, and the fiber cut comments hit the list about 4 minutes before your message. The internet as a whole has had substantially less Reserve capacity, as the power generation people like to put it, for about the last month, and wh

Re: [outages] Centurylink Packet Loss - Socal

The occasional batch of dropped packets in Arkansas, old L3 network. On Tue, Apr 21, 2020 at 11:22 AM David Hubbard via Outages < outages@outages.org> wrote: > We're seeing issues in both Tampa and Las Vegas with them. > > On 4/21/20, 11:52 AM, "Outages on behalf of Douglas K. Rand via Outages"

Re: [outages] US West Coast Massive Tier 1 Backbone Issues

I was hoping somebody was going to climb up to 30,000 feet on this, because it sort of felt to me like it was pervasive as well. I encourage further analysis on this point, and apologize to anybody who would prefer that it be on the discussion list - I think it actually belongs here. On April

Re: [outages] Centurylink Packet Loss - Socal

Got this update from CL. 04/21/2020 16:07:37 GMT - *** CASCADED EXTERNAL NOTES 21-Apr-2020 16:07:34 GMT From CASE: 18573666 - Event IP NOC reports that services are impacted by multiple fiber cuts that are impacting the CenturyLink network. Field Operations and all necessary repair personnel hav

Re: [outages] COX outage?

We saw similar Loss between COX and level 3 in our west coast DC and have migrated off COX links for now and are not seeing the loss. From: Outages On Behalf Of Andree Toonk via Outages Sent: Tuesday, April 21, 2020 8:51 AM To: Sam Thomas Cc: outages@outages.org Subject: Re: [outages] COX ou

Re: [outages] Centurylink Packet Loss - Socal

We're seeing issues in both Tampa and Las Vegas with them. On 4/21/20, 11:52 AM, "Outages on behalf of Douglas K. Rand via Outages" wrote: We have been seeing CL packet loss since around 9:30am CDT. Mostly on their Level3 network, seems to be between Atlanta and Minneapolis for u

[outages] US West Coast Massive Tier 1 Backbone Issues

Hey All, There's a few threads on this already, but they all call out specific ISPs when this trouble isn't limited to any one last-mile provider. Nearly all tier 1 transit providers are experiencing loss on the US West Coast today. Some examples: Comcast/Level3 peering point in Seattle: [image:

Re: [outages] Centurylink Packet Loss - Socal

We are also seeing ~10% loss and ~100ms extra latency on CTL; between Spokane, WA and DC. It does appear to be occurring on the old L3 network. --Eric -Original Message- From: Outages On Behalf Of Douglas K. Rand via Outages Sent: Tuesday, April 21, 2020 8:43 AM To: outages@outages.org

Re: [outages] COX outage?

We are seeing major packet loss on Level3 at various of our west coast north America locations, also Denver. Guessing it's related... Cheers  Andree Sam Thomas via Outages wrote on 2020-04-21 8:35 AM: A number of our remote users and some of our remote offices on COX are having some extreme

Re: [outages] Centurylink Packet Loss - Socal

We have been seeing CL packet loss since around 9:30am CDT. Mostly on their Level3 network, seems to be between Atlanta and Minneapolis for us. Perhaps a broader CL/L3 issue? On 4/21/20 10:36 AM, Samuel Jackson via Outages wrote: Anyone experiencing CL internet packet loss in the San Diego area

[outages] COX outage?

A number of our remote users and some of our remote offices on COX are having some extreme difficulties with their VPN connections. One of the users called COX and was informed that there is a "major outage". Could someone shed some light on the scope of this "major outage"? (e.g. geographic, pe

[outages] Centurylink Packet Loss - Socal

Anyone experiencing CL internet packet loss in the San Diego area? We started seeing a spike around 7am PDT. Pings to multiple remote sites show around 40% loss. Downdetector also shows some activity. Thanks, Bobin ___ Outages mailing list Outages@outag

[outages] Packet Loss from Switch DC in LAS

We are experiencing high latency and packet loss on the egress side of calls into and out of our Las Vegas Switch data center. Ticket with Switch has been opened and we’ve diverted all media traffic away from the data center until this is resolved. Rob Whitlatch Director of DevOps White Label