What in depth analysis have you seen? Seems to me, this was a failure in a known maintenance activity, and they simply disconnected the devices under maintenance from the network.
Shane On Mon, Jul 11, 2022 at 5:41 AM Jon Sands <fohdee...@gmail.com> wrote: > Given the outage was so bad it was disrupting select E911 services > nationwide for something like 24+ hours, it's great to see such an in depth > analysis and plan of action to prevent such things in the future. bravo > rogers > > On 7/10/2022 2:55 PM, L F wrote: > > fyi - see BOLD. > > A Message from Rogers President and CEO > > > Dear Valued Customer, > > As you know, we experienced a service outage across the Rogers, Fido, > Chatr and Cityfone wireless networks on Friday. > > I am reaching out to share that our services have been restored, and our > networks and systems are close to fully operational. Our technical teams > are continuing to monitor for any remaining intermittent issues. I also > want to outline an action plan we are putting in place to address what > happened. > > I want to share what we know about what happened on Friday. We now believe > *we’ve > narrowed the cause to a network system failure following a maintenance > update in our core network, which caused some of our routers to > malfunction. We disconnected the specific equipment and redirected traffic,* > which allowed our network and services to come back online over time as we > managed traffic volumes returning to normal levels. > > > > On Sat, Jul 9, 2022 at 9:09 PM L F <liz.faze...@gmail.com> wrote: > >> >> Lest we ALL MOVE ON…. >> >> >> Yes he said >> >> “RETARD” = >> >> The delay in processing the current status of a Situation… >> >> >> Lets move on to bigger n scarier Post mortems: >> >> >> Was This or Was this NOT >> >> A CYBERATTACK ??? >> >> Done. >> >> Lets deal with REAL threats not PC references. >> >> Time to evaluate n re eval Our NTS!! >> >> Veni vidi vici 2022 >> >> >> >> >> On Sat, Jul 9, 2022 at 5:01 PM Eric Kuhnke <eric.kuh...@gmail.com> wrote: >> >>> Can we have a discussion with the list admins about a list member >>> appending a threat of violence to their outbound emails? Whether serious >>> or not. >>> >>> Does this person need directions to some local mental healthcare >>> resources? >>> >>> >>> On Sat, 9 Jul 2022 at 08:48, Keith Medcalf <kmedc...@dessus.com> wrote: >>> >>>> >>>> >I can't either, but the reality right now seems to be that 911 calls >>>> are >>>> >failing for anyone on a Rogers cellphone. >>>> >>>> This is par for the course. These people chose to deal with Rogers >>>> despite knowing the consequences. It is like if you bought a Rogers >>>> Snowblower and it did not work. That would mean that people who bought the >>>> Rogers Snowblower will not be using it to get rid of the snow that is >>>> preventing them from leaving their house. >>>> >>>> Mutatis mutandis when Rogers is down things that are Rogers dependent >>>> will not work. >>>> >>>> Some people are so retarded it is astonishing! >>>> >>>> -- >>>> (CAUTION) You are advised that if you attack my person or property, you >>>> will be put down in accordance with the provisions of section 34 & 35 of >>>> the Criminal Code respectively. If you are brandishing (or in possession) >>>> of a weapon then lethal force will be applied to your person in accordance >>>> with the law. This means that your misadventures may end in your death. >>>> Consider yourself cautioned and govern your actions appropriately. >>>> >>>> >-----Original Message----- >>>> >From: NANOG <nanog-bounces+kmedcalf=dessus....@nanog.org> On Behalf Of >>>> >Eric Kuhnke >>>> >Sent: Friday, 8 July, 2022 13:34 >>>> >To: jim deleskie <deles...@gmail.com> >>>> >Cc: NANOG list <nanog@nanog.org> >>>> >Subject: Re: Rogers Outage Canada >>>> > >>>> > >>>> >I have seen anecdotal reports that the mobile network is in a half >>>> broken >>>> >state that phones remain registered to, so a 911 call will attempt and >>>> >then fail. >>>> > >>>> > >>>> >This is unlike what would happen if you had a US/Canada cellphone with >>>> >battery power but no SIM card in it that would search for any available >>>> >network in RF range for a 911 call if needed. >>>> > >>>> > >>>> >On Fri, 8 Jul 2022 at 12:31, jim deleskie <deles...@gmail.com >>>> ><mailto:deles...@gmail.com> > wrote: >>>> > >>>> > >>>> > i cant see BGP taking out SS7. >>>> > >>>> > -jim >>>> > >>>> > On Fri, Jul 8, 2022 at 2:45 PM Snowmobile2004 >>>> ><greenjosh6...@gmail.com <mailto:greenjosh6...@gmail.com> > wrote: >>>> > >>>> > >>>> > According to Cloudflare Radar >>>> ><https://radar.cloudflare.com/asn/812?date_filter=last_24_hours> , >>>> Rogers >>>> >BGP announcements spiked massively to levels 536,777% higher than >>>> normal >>>> >(343,601 vs 64 normally) just minutes before the outage. I would not be >>>> >surprised if this happened to be the culprit. >>>> > >>>> > Regards, >>>> > Josh Green >>>> > >>>> > On Fri, Jul 8, 2022 at 2:19 PM Andrew Paolucci via NANOG >>>> ><nanog@nanog.org <mailto:nanog@nanog.org> > wrote: >>>> > >>>> > >>>> > In the early hours of the morning around 2-3am >>>> my modem >>>> >got hit with a configuration update that caused a DHCP release that >>>> >wasn't renewed for about two hours, after rollback the connection was >>>> >fine for 3 hours before this network wide outage. >>>> > >>>> > >>>> > Maybe a failed night time update was attempted >>>> again >>>> >during office hours, I've heard daytime guys are still WFH and night >>>> >shift is in building. >>>> > >>>> > >>>> > I expect we'll never get a real explanation. >>>> Rogers is >>>> >notorious for withholding any type of helpful or technical information. >>>> > >>>> > >>>> > Sent from my inoperable Rogers Mobile via >>>> emergency eSIM. >>>> > >>>> > >>>> > Regards, >>>> > >>>> > Andrew Paolucci >>>> > -------- Original Message -------- >>>> > On Jul. 8, 2022, 1:48 p.m., Jay Hennigan < >>>> j...@west.net >>>> ><mailto:j...@west.net> > wrote: >>>> > >>>> > >>>> > On 7/8/22 07:44, Robert DeVita wrote: > >>>> Does anyone >>>> >have information on a widespread Rogers outage in Canada. I > have >>>> >customers with multiple sites down. There's discussion on the Outages >>>> >mailing list. Seems widespread, affecting all services, mobile, voice, >>>> >Internet. No cause or ETR posted yet. -- Jay Hennigan - j...@west.net >>>> ><mailto:j...@west.net> Network Engineering - CCIE #7880 503 897-8550 - >>>> >WB6RDV >>>> > >>>> > >>>> > >>>> > -- >>>> > >>>> > Josh Green. >>>> >>>> >>>> >>>> > > > -- > Jon Sands > MFI Labshttps://fohdeesha.com/ > >
