My apologies if this message breaks the mailing list, I have never replied to one of these before.
I work for a rural ISP, we do not use CenturyLink as an upstream; however, some of our customers are reporting issues since the outage. Management has tasked my office with putting together a timeline/details about the outage to brief upper management. Is there a detailed timeline of events documenting this outage? Thanks, A On Fri, Dec 28, 2018 at 12:13 PM <outages-requ...@outages.org> wrote: > Send Outages mailing list submissions to > outages@outages.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://puck.nether.net/mailman/listinfo/outages > or, via email, send a message with subject or body 'help' to > outages-requ...@outages.org > > You can reach the person managing the list at > outages-ow...@outages.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Outages digest..." > > > Today's Topics: > > 1. Re: CenturyLink Outages this morning (Aaron C. de Bruyn) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 28 Dec 2018 09:12:03 -0800 > From: "Aaron C. de Bruyn" <aa...@heyaaron.com> > To: Dom Colangelo <dom.colang...@simmons.edu> > Cc: Bob Antia <an...@ba14.com>, outages@outages.org > Subject: Re: [outages] CenturyLink Outages this morning > Message-ID: > <CAEE+rGqaysPos= > vb_qts++cbs1bnurzxclig0ibmntlqcvv...@mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Several times late last night we had our weather alert radio activate and > broadcast an unintelligible message about 911 being down. It wasn't > reception problems. The cadence of the dumb computer voice made it nearly > impossible to understand. > > https://photos.app.goo.gl/M4spwKAkKPnoSrJw9 > > The alert came through ~5 hours after 911 went offline everywhere. > > -A > > > On Fri, Dec 28, 2018, 08:44 Dom Colangelo via Outages <outages@outages.org > wrote: > > > Received a WEA from MEMA in Boston, MA noting that 911 is down for > > cellular users. > > > > Dom Colangelo > > > > Senior Windows Systems Administrator > > > > Infrastructure and Online Services > > > > Simmons University > > > > 617.521.2669 > > > > > > On Fri, Dec 28, 2018 at 9:56 AM Bob Antia via Outages < > outages@outages.org> > > wrote: > > > >> Massachusetts emergency management agency (MEMA) sent this out this > >> morning and followed up at 0736 that the issues had been resolved. > >> > >> > >> *Massachusetts Emergency Management Agency* > >> ------------------------------ > >> > >> *Massachusetts Emergency Management Agency* > >> > >> *Situational Awareness Statement* > >> > >> *December 28, 2018 7:00 AM* > >> > >> > >> > >> *Re: Cellular 911 Experiencing Technical Difficulties and Sporadic > >> Outages* > >> > >> > >> > >> The Massachusetts Emergency Management Agency is sharing the following > >> information from the Commonwealth Fusion Center (CFC) with our local > >> partners and stakeholders. MEMA also is in touch with the State 911 > >> Department. > >> > >> > >> > >> *From the Commonwealth Fusion Center* > >> > >> > >> > >> *Situation:* > >> > >> A nationwide outage affecting CenturyLink, an Internet communications > >> company, is impacting wireless 911 call capability in various parts of > the > >> country. We believe Massachusetts has been impacted by the outage and > urge > >> residents using cellular phones, if necessary, to contact emergency > >> first-responders by using the full 10-digit number of a public safety > >> answering point emergency call center (PSAP) or their local police > >> department. > >> > >> > >> > >> The Massachusetts State Police operate three regional PSAPs and a fourth > >> is staffed by State 911 at MSP headquarters in Framingham. > >> Residents using a cell phone to call for emergency services should first > >> try to dial 911, and if it does not work, instead call one of these four > >> 10-digit numbers: > >> Shelburne Control PSAP 413-625-8200 > >> New Braintree Control PSAP 508-867-1170 > >> Northampton Control PSAP 413-586-1508 > >> Framingham PSAP 508-820-2121. > >> > >> > >> Residents using cell phones may also, if dialing 911 does not work, may > >> call the 10-digit general number of their local police department. They > may > >> also call the 10-digit number of a State Police Barracks in their area > >> (search for barracks location and phone numbers on the MSP web site at > >> www.mass.gov and search for State Police locations). > >> Other agencies operate additional PSAPs in Massachusetts beyond those > >> listed above. > >> > >> > >> > >> The outage seems to only be affecting wireless 911 calls; landline calls > >> to 911 appear to be working. > >> > >> > >> CenturyLink has indicted it is working on repairing what the company > >> called a ?network element? that is causing the problem. > >> > >> > >> > >> *MEMA Operations* > >> > >> The State Emergency Operations Center (SEOC) is currently operating at > >> Level 1 (Steady State Monitoring). MEMA will continue to monitor the > >> situation and will disseminate additional Situational Awareness > Statements > >> as necessary. Cities and towns wishing to share information or request > >> assistance should contact their MEMA Regional office or MEMA?s 24/7 > >> communications center at 508-820-2000. > >> > >> *Stay Informed:* > >> > >> Utilize *Massachusetts Alerts* to receive emergency notifications and > >> information from the Massachusetts Emergency Management Agency and the > >> National Weather Service. *Massachusetts Alerts* is a free app that is > >> available for Android and iPhones. To learn more about *Massachusetts > >> Alerts*, and for information on how to download the free app onto your > >> smartphone, visit: > >> http://www.mass.gov/service-details/massachusetts-alerts-smartphone-app > . > >> > >> Utilize MEMA?s real-time power outage viewer to stay informed about > >> current power outages in your community and region, and across the > state, > >> including information from utility companies about restoration times: > >> http://mema.mapsonline.net/public.html > >> > >> Utilize MEMA?s live weather radar and forecasting tools: > >> http://www.mass.gov/map-resources > >> > >> > >> *Online Resources:*For additional information and resources, visit: > >> Massachusetts Emergency Management Agency at www.mass.gov/mema > >> MEMA's Facebook page: http://www.facebook.com/MassachusettsEMA > >> MEMA Twitter: @MassEMA <http://twitter.com/massema> > >> Federal Emergency Management Agency at www.fema.gov > >> National Weather Service Boston/Norton at www.weather.gov/boston > >> National Weather Service Albany, NY at www.weather.gov/albany > >> National Weather Service Weather Prediction Center: > >> http://www.wpc.ncep.noaa.gov > >> National Weather Service Storm Prediction Center: > >> http://www.spc.noaa.gov/ > >> Mass211 at www.mass211.org > >> > >> > >> > >> > >> ------------------------------ > >> > >> This message has been sent from MEMA's ESF Team Listserv. > >> To unsubscribe from this listserv, call MEMA Operations at (508) > 820-2000. > >> > >> +1.781.300.0000 > >> > >> > >> On Fri, Dec 28, 2018 at 9:52 AM Matt Brennan via Outages < > >> outages@outages.org> wrote: > >> > >>> Most public safety agencies in Massachusetts posted on Facebook and > >>> Twitter between 6 AM and 8 AM EST about wireless 911 outages caused by > >>> CenturyLink. It was affecting wireless routing, so some calls were > being > >>> misrouted to other towns, or even other states; some callers were > getting > >>> reorder tones or other error messages. The problem was reported to be > >>> resolved prior to 8 AM EST. > >>> > >>> Seeing reports from PSAP's that the issue started again in the last 15 > >>> minutes. > >>> > >>> This is wireless only. Landline 911 calls were / are not affected. > >>> > >>> > >>> On Fri, Dec 28, 2018 at 9:32 AM Mel Beckman via Outages < > >>> outages@outages.org> wrote: > >>> > >>>> Don't be a fear monger. I searched Facebook and found *three* posts > >>>> about 911 outages, all of them simply reposts of local news stories. > >>>> > >>>> Facebook is not "blowing up". > >>>> > >>>> Sheesh! > >>>> > >>>> [image: image1.PNG] > >>>> > >>>> -mel via cell > >>>> > >>>> -mel via cell > >>>> > >>>> On Dec 28, 2018, at 5:23 AM, Josh Luthman via Outages < > >>>> outages@outages.org> wrote: > >>>> > >>>> Facebook is blowing up with 911 outages and claiming it's CenturyLink > >>>> to blame. > >>>> > >>>> Josh Luthman > >>>> Office: 937-552-2340 > >>>> Direct: 937-552-2343 > >>>> 1100 Wayne St > >>>> Suite 1337 > >>>> Troy, OH 45373 > >>>> > >>>> On Fri, Dec 28, 2018, 2:09 AM Erik Sundberg via Outages < > >>>> outages@outages.org wrote: > >>>> > >>>>> Maybe light at the end of the tunnel... > >>>>> > >>>>> > >>>>> > >>>>> latest we got from Centurylink was at 10:45pm CST. > >>>>> > >>>>> Our engineers and technicians have identified the network element > that > >>>>> has affected our customer services. Services are restoring, and the > current > >>>>> estimated time for full recovery is four hours. > >>>>> > >>>>> CenturyLink will be conducting an extensive post-incident > >>>>> investigation and root cause analysis to provide follow-up > information to > >>>>> our customers impacted by this event. We will provide one more Sales > Alert > >>>>> from this address once services are fully restored. > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> *From:* Rich Breton <richdbre...@gmail.com> > >>>>> *Sent:* Thursday, December 27, 2018 2:59 PM > >>>>> *To:* Christopher Conley <ccon...@forsmarshgroup.com> > >>>>> *Cc:* Erik Sundberg <esundb...@nitelusa.com>; outages@outages.org > >>>>> *Subject:* Re: [outages] CenturyLink Outages this morning > >>>>> > >>>>> > >>>>> > >>>>> I understand that the transport waves went out first, and then they > >>>>> try to reroute to something in Arizona that begin introducing all of > the > >>>>> broadcast traffic. How legit that is I?m not sure > >>>>> > >>>>> Sent from my iPhone > >>>>> > >>>>> > >>>>> On Dec 27, 2018, at 12:58 PM, Christopher Conley via Outages < > >>>>> outages@outages.org> wrote: > >>>>> > >>>>> > Finally received something useful from centurylink? I have more > >>>>> questions now like how broadcast traffic effect 4x Transport waves. > >>>>> > >>>>> > >>>>> > >>>>> Ineptitude. > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> Christopher Conley > >>>>> > >>>>> Systems Administrator | Fors Marsh Group > >>>>> > >>>>> Arlington, VA > >>>>> > >>>>> > >>>>> > >>>>> *From:* Outages <outages-boun...@outages.org> *On Behalf Of *Erik > >>>>> Sundberg via Outages > >>>>> *Sent:* Thursday, December 27, 2018 2:46 PM > >>>>> *To:* outages@outages.org > >>>>> *Subject:* Re: [outages] CenturyLink Outages this morning > >>>>> > >>>>> > >>>>> > >>>>> Finally received something useful from centurylink? I have more > >>>>> questions now like how broadcast traffic effect 4x Transport waves. > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> Update: > >>>>> > >>>>> On December 27, 2018 at 02:40 GMT, CenturyLink identified a service > >>>>> impact in New Orleans, LA. The NOC is engaged and investigating in > order to > >>>>> isolate the cause. Field Operations were engaged and dispatched for > >>>>> additional investigations. Tier IV Equipment Vendor Support was later > >>>>> engaged. During cooperative troubleshooting a device in San Antonio, > TX was > >>>>> isolated from the network as it was seeming to broadcast traffic > consuming > >>>>> capacity, which seemed to alleviate some impact. Investigations > remained > >>>>> ongoing. Following the isolation of the San Antonio, TX device > >>>>> troubleshooting efforts focused on additional sites that teams were > >>>>> remotely unable to troubleshoot. Field Operations were dispatched to > sites > >>>>> in Kansas City, MO, Atlanta, GA, New Orleans, LA and Chicago, IL. > Tier IV > >>>>> Equipment Vendor Support continued to investigate the equipment logs > to > >>>>> further assist with isolation. Once visibility was restored to the > site in > >>>>> Kansas City, MO a filter was applied to the equipment to further > alleviate > >>>>> the impact observed. All of the necessary troubleshooting teams in > >>>>> cooperation with Tier IV Equipment Vendor Support are working to > restore > >>>>> remote visibility to the remaining sites at this time. We understand > how > >>>>> important these services are to our clients and the issue has been > >>>>> escalated to the highest levels within CenturyLink Service Assurance > >>>>> Leadership. > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> *From:* Outages <outages-boun...@outages.org> *On Behalf Of *Bruce > >>>>> Freshwater via Outages > >>>>> *Sent:* Thursday, December 27, 2018 1:24 PM > >>>>> *To:* outages@outages.org > >>>>> *Subject:* Re: [outages] CenturyLink Outages this morning > >>>>> > >>>>> > >>>>> > >>>>> CenturyLink finally acknowledged: > >>>>> > >>>>> Great Lakes Region CenturyLink Partners/Agents: > >>>>> > >>>>> > >>>>> > >>>>> FYI. Our network is experiencing a disruption affecting customer > >>>>> services. Customers are currently unable to generate tickets through > the > >>>>> Control Center portal. > >>>>> > >>>>> > >>>>> > >>>>> We know how important these services are to our customers, and we are > >>>>> working to restore services as quickly as possible. > >>>>> > >>>>> > >>>>> > >>>>> Our technical teams are working to resolve the issue, and we will > >>>>> provide updates as the situation progresses. > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> -bruce > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> On Thu, Dec 27, 2018 at 10:16 AM -0500, "Erik Sundberg via Outages" < > >>>>> outages@outages.org> wrote: > >>>>> > >>>>> So it's a bad day for centurylink. > >>>>> > >>>>> > >>>>> > >>>>> The lady who answered the phone for there NOC said that we are having > >>>>> a nation wide outage and we are unable to open tickets at this time. > Please > >>>>> call be in 60-90 mins. > >>>>> > >>>>> > >>>>> > >>>>> She had no info on what was going on > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> ------------------------------ > >>>>> > >>>>> *From:* Michael Bolton <mbol...@holmeselectricsecurity.com> > >>>>> *Sent:* Thursday, December 27, 2018 8:55:03 AM > >>>>> *To:* Erik Sundberg; outages@outages.org > >>>>> *Subject:* RE: CenturyLink Outages this morning > >>>>> > >>>>> > >>>>> > >>>>> Seeing the same in NC. Circuits are up but the routing between POPs > >>>>> has changed and we are seeing higher than normal latency. > >>>>> > >>>>> > >>>>> > >>>>> VoIP portal is down also. Allows you to login but you cannot make any > >>>>> changes or view any details. > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> *From:* Outages <outages-boun...@outages.org> *On Behalf Of *Erik > >>>>> Sundberg via Outages > >>>>> *Sent:* Thursday, December 27, 2018 9:45 AM > >>>>> *To:* outages@outages.org > >>>>> *Subject:* [outages] CenturyLink Outages this morning > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> *CAUTION:* This email originated from outside of the organization. Do > >>>>> not click links or open attachments unless you recognize the sender > and > >>>>> know the content is safe. > >>>>> > >>>>> We are seeing a lot of centurylink circuits and services down this > >>>>> morning. > >>>>> > >>>>> > >>>>> > >>>>> Centurylink Control Center portal comes up but auth failing unable to > >>>>> login > >>>>> > >>>>> > >>>>> > >>>>> Waves down or bouncing > >>>>> > >>>>> Denver - Seattle (Bouncing) > >>>>> > >>>>> Denver - Chicago (Bouncing) > >>>>> > >>>>> New York - Los Angeles (Down) > >>>>> > >>>>> Chicago - Atlanta (Down) > >>>>> > >>>>> > >>>>> ENNI Down in Denver (Down) > >>>>> > >>>>> > >>>>> > >>>>> Still tying to open tickets with them there portal is not working and > >>>>> stuck on hold right now. > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> ------------------------------ > >>>>> > >>>>> > >>>>> CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, > >>>>> files or previous e-mail messages attached to it may contain > confidential > >>>>> information that is legally privileged. If you are not the intended > >>>>> recipient, or a person responsible for delivering it to the intended > >>>>> recipient, you are hereby notified that any disclosure, copying, > >>>>> distribution or use of any of the information contained in or > attached to > >>>>> this transmission is STRICTLY PROHIBITED. If you have received this > >>>>> transmission in error please notify the sender immediately by > replying to > >>>>> this e-mail. You must destroy the original transmission and its > attachments > >>>>> without reading or saving in any manner. Thank you. > >>>>> > >>>>> IMPORTANT NOTICE: This e-mail message is intended to be received only > >>>>> by persons entitled to receive the confidential information it may > contain. > >>>>> E-mail messages to clients of Holmes Security Systems may contain > >>>>> information that is confidential and legally privileged. Please do > not > >>>>> read, copy, forward, or store this message unless you are an intended > >>>>> recipient of it. If you have received this message in error, please > forward > >>>>> it to the sender and delete it completely from your computer system. > >>>>> > >>>>> > >>>>> ------------------------------ > >>>>> > >>>>> > >>>>> CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, > >>>>> files or previous e-mail messages attached to it may contain > confidential > >>>>> information that is legally privileged. If you are not the intended > >>>>> recipient, or a person responsible for delivering it to the intended > >>>>> recipient, you are hereby notified that any disclosure, copying, > >>>>> distribution or use of any of the information contained in or > attached to > >>>>> this transmission is STRICTLY PROHIBITED. If you have received this > >>>>> transmission in error please notify the sender immediately by > replying to > >>>>> this e-mail. You must destroy the original transmission and its > attachments > >>>>> without reading or saving in any manner. Thank you. > >>>>> > >>>>> > >>>>> ------------------------------ > >>>>> > >>>>> > >>>>> CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, > >>>>> files or previous e-mail messages attached to it may contain > confidential > >>>>> information that is legally privileged. If you are not the intended > >>>>> recipient, or a person responsible for delivering it to the intended > >>>>> recipient, you are hereby notified that any disclosure, copying, > >>>>> distribution or use of any of the information contained in or > attached to > >>>>> this transmission is STRICTLY PROHIBITED. If you have received this > >>>>> transmission in error please notify the sender immediately by > replying to > >>>>> this e-mail. You must destroy the original transmission and its > attachments > >>>>> without reading or saving in any manner. Thank you. > >>>>> > >>>>> _______________________________________________ > >>>>> Outages mailing list > >>>>> Outages@outages.org > >>>>> https://puck.nether.net/mailman/listinfo/outages > >>>>> > >>>>> > >>>>> ------------------------------ > >>>>> > >>>>> CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, > >>>>> files or previous e-mail messages attached to it may contain > confidential > >>>>> information that is legally privileged. If you are not the intended > >>>>> recipient, or a person responsible for delivering it to the intended > >>>>> recipient, you are hereby notified that any disclosure, copying, > >>>>> distribution or use of any of the information contained in or > attached to > >>>>> this transmission is STRICTLY PROHIBITED. If you have received this > >>>>> transmission in error please notify the sender immediately by > replying to > >>>>> this e-mail. You must destroy the original transmission and its > attachments > >>>>> without reading or saving in any manner. Thank you. > >>>>> _______________________________________________ > >>>>> Outages mailing list > >>>>> Outages@outages.org > >>>>> https://puck.nether.net/mailman/listinfo/outages > >>>>> > >>>> _______________________________________________ > >>>> Outages mailing list > >>>> Outages@outages.org > >>>> https://puck.nether.net/mailman/listinfo/outages > >>>> > >>>> _______________________________________________ > >>>> Outages mailing list > >>>> Outages@outages.org > >>>> https://puck.nether.net/mailman/listinfo/outages > >>>> > >>> _______________________________________________ > >>> Outages mailing list > >>> Outages@outages.org > >>> https://puck.nether.net/mailman/listinfo/outages > >>> > >> _______________________________________________ > >> Outages mailing list > >> Outages@outages.org > >> https://puck.nether.net/mailman/listinfo/outages > >> > > _______________________________________________ > > Outages mailing list > > Outages@outages.org > > https://puck.nether.net/mailman/listinfo/outages > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > https://puck.nether.net/pipermail/outages/attachments/20181228/e8c66add/attachment.html > > > -------------- next part -------------- > A non-text attachment was scrubbed... > Name: image1.PNG > Type: image/png > Size: 55125 bytes > Desc: not available > URL: < > https://puck.nether.net/pipermail/outages/attachments/20181228/e8c66add/attachment.png > > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > Outages mailing list > Outages@outages.org > https://puck.nether.net/mailman/listinfo/outages > > > ------------------------------ > > End of Outages Digest, Vol 127, Issue 31 > **************************************** > -- *Aaron K. Henderson* Agri-Valley Communications, Inc. Network Technician *O:* (989) 453-4400 <9894534400> *C:* (989) 551-0060 <9895510060> *netad...@avci.net <netad...@avci.net> *
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
