Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

Hey All, This sounds much like what happened in Canada with Rogers – https://en.wikipedia.org/wiki/2022_Rogers_Communications_outage The same words seem to be coming from the top  From: AusNOG on behalf of DaZZa Date: Wednesday, November 8, 2023 at 1:10 PM To: Jake Anderson Cc:

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

The ABC's rolling coverage reports "Some services across fixed and mobile are now gradually being restored.", but it may take a "few hours" for everything to recover. Now we just get to wait for the report on the outage. it oughta be a doozy! D On Wed, 8 Nov 2023 at 12:15, Jake Anderson wrote:

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

Those who believe it's a BGP issue might want to review https://www.cidr-report.org/cgi-bin/plota?file=%2fvar%2fdata%2fbgp%2fas2.0%2fbgp%2dactive%2etxt=Active%20BGP%20entries%20%28FIB%29=Active%20BGP%20entries%20%28FIB%29=step This shows a global table getting dangerously close to a 1024K

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

They weren't appearing earlier so I'd say it's new. Smells like BGP has shat itself. - ​CH From: AusNOG on behalf of Tim Malone Sent: Wednesday, November 8, 2023 12:16 PM To: Nick Jenkin Cc: ausnog@lists.ausnog.net Subject: Re: [AusNOG] Optus downtime chat +

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

Cloudflare is also reporting BGP origin hijack warnings: https://radar.cloudflare.com/as7474?dateRange=2d (unsure if related though) On Wed, 8 Nov 2023 at 11:10, Nick Jenkin wrote: > Just a few announcements... > https://radar.cloudflare.com/traffic/as4804?dateRange=2d > > On Wed, 8 Nov 2023

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

Looks like they may getting some services back https://radar.cloudflare.com/as4804?dateStart=2023-11-07=2023-11-07 On 8/11/2023 9:26 am, francisfi...@mailup.net wrote: Morning all, Hope the chaos isn't too hard on your work/family. I have had trouble with a couple of SMS verifications coming

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

I’m wondering if it is leaking over to Vodafone as International Roaming has failed for me here in Prague at about the same time. All I’m getting is “no servíce” even after forcing the carrier to Vodafone CZ which I’ve been using the last 8 days. -- Mark Andrews > On 8 Nov 2023, at 01:10,

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

Just a few announcements... https://radar.cloudflare.com/traffic/as4804?dateRange=2d On Wed, 8 Nov 2023 at 11:06, Ben Buxton wrote: > > > On Wed, 8 Nov 2023 at 10:14, DaZZa wrote: > >> Yeah, I'd be willing to bet that it's a change which wasn't thoroughly >> tested before being rolled out, and

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

On Wed, 8 Nov 2023 at 10:14, DaZZa wrote: > Yeah, I'd be willing to bet that it's a change which wasn't thoroughly > tested before being rolled out, and which had an inadequate backout > plan. > Also, "Our on-site technician is actively prioritising establishing a console connection.". I mean

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

I loved the scramble to avoid saying "We f* up". And the dodge when asked "How many staff are on-shore and how manby overseas". I laughed She's a talking head. I didn't expect any information from her - what I took out of it is that 7 hours after this started, they *still* don't know what

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

Gotta say, that interview didn't give me any confidence at all. It makes me wonder if the CEO even knows what Optus does as its core business, or has had any sort of media training on how to communicate with customers and the public. "We'd like the media to help by directing people to our

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

I happened to be looking at my phone when it happened. It said words to the effect of 'the carrier has turned off mobile data for a short time" with a little triangle symbol that I've never seen before.I guess it wasn't wrong depending on how you define shortSent from my Galaxy

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

Maybe they were annoyed at glen maxwell getting 200 Get Outlook for iOS From: AusNOG on behalf of francisfi...@mailup.net Sent: Wednesday, November 8, 2023 6:26:26 AM To: AusNOG@lists.ausnog.net Subject: [AusNOG] Optus downtime chat +

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

https://www.abc.net.au/news/2023-11-08/optus-outage-live-blog/103076996 Get Outlook for iOS From: AusNOG on behalf of DaZZa Sent: Wednesday, November 8, 2023 6:39:22 AM To: francisfi...@mailup.net Cc: AusNOG@lists.ausnog.net Subject:

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

On Wed, 8 Nov 2023 at 09:29, Joseph Goldman wrote: > > I would also have to imagine the outage is a bad config pushed out - > surely a hardware failure or fibre cut wouldn't cause such a big outage. > Bad day to have your name on the last push :/. > In a blameless culture, I'l probably consider

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

Yeah, I'd be willing to bet that it's a change which wasn't thoroughly tested before being rolled out, and which had an inadequate backout plan. Interestingly, my Optus mobile actually had a valid connection for a short time - wasn't able to actually DO anything, but was connected to the OPtus

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

The 4am Wednesday morning outage start looks suspiciously like a firmware upgrade window. I note that Optus devices where I am are showing "SoS" which indicates the tower is unable to reach the location register, which presumably is on a private network and indicative of a pretty major fault

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

The Optus hamster finally died of old age. I would suggest your SMS issues would be caused by whoever is issuing the SMS using Optus - not so much by the Telstra end receiving it. Anecdotally, Optus enterprise/wholesale appears to be still functional - at least my link appears to be working fine

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

This was the update a bit after 7:30: The suspected root cause of the issue lies with the route reflectors, which are currently handling an excessive number of routes, leading to session shutdown and a complete traffic halt. Our on-site technician is actively prioritising establishing a console

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

Was actually wondering why there wasn't a thread. My guess would be for 2FA SMS's not coming through, that the SMS provider the software/site is using relies on Optus to send the message. I would also have to imagine the outage is a bad config pushed out - surely a hardware failure or fibre

[AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

Morning all, Hope the chaos isn't too hard on your work/family. I have had trouble with a couple of SMS verifications coming through to me, my Telstra number. Is this related? Any general banter around the downtime would be fine too - looks like it all began at 4.07am AEDT? Cheers --