[AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[francisfides]

Morning all,
Hope the chaos isn't too hard on your work/family. 
I have had trouble with a couple of SMS verifications coming through to me, my 
Telstra number. Is this related?

Any general banter around the downtime would be fine too - looks like it all 
began at 4.07am AEDT?

Cheers

-- 
  
  francisfi...@mailup.net
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[Joseph Goldman]

Was actually wondering why there wasn't a thread.

My guess would be for 2FA SMS's not coming through, that the SMS 
provider the software/site is using relies on Optus to send the message.


I would also have to imagine the outage is a bad config pushed out - 
surely a hardware failure or fibre cut wouldn't cause such a big outage. 
Bad day to have your name on the last push :/.


-- Original Message --
From: francisfi...@mailup.net
To: AusNOG@lists.ausnog.net
Sent: 8/11/2023 9:26:26 AM
Subject: [AusNOG] Optus downtime chat + affecting SMS verification to 
Telstra?



Morning all,
Hope the chaos isn't too hard on your work/family.
I have had trouble with a couple of SMS verifications coming through to me, my 
Telstra number. Is this related?

Any general banter around the downtime would be fine too - looks like it all 
began at 4.07am AEDT?

Cheers

--

francisfi...@mailup.net
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog



___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[Shaun Coffey]

This was the update a bit after 7:30:

The suspected root cause of the issue lies with the route reflectors, which
are currently handling an excessive number of routes, leading to session
shutdown and a complete traffic halt. Our on-site technician is actively
prioritising establishing a console connection. Rest assured that said
technician is also being provided additional technical support remotely.
Kindly expect next update to be provided by 9AM AEDT if not sooner.


On Wed, 8 Nov 2023 at 9:29 am, Joseph Goldman  wrote:

> Was actually wondering why there wasn't a thread.
>
> My guess would be for 2FA SMS's not coming through, that the SMS
> provider the software/site is using relies on Optus to send the message.
>
> I would also have to imagine the outage is a bad config pushed out -
> surely a hardware failure or fibre cut wouldn't cause such a big outage.
> Bad day to have your name on the last push :/.
>
> -- Original Message --
> From: francisfi...@mailup.net
> To: AusNOG@lists.ausnog.net
> Sent: 8/11/2023 9:26:26 AM
> Subject: [AusNOG] Optus downtime chat + affecting SMS verification to
> Telstra?
>
> >Morning all,
> >Hope the chaos isn't too hard on your work/family.
> >I have had trouble with a couple of SMS verifications coming through to
> me, my Telstra number. Is this related?
> >
> >Any general banter around the downtime would be fine too - looks like it
> all began at 4.07am AEDT?
> >
> >Cheers
> >
> >--
> >
> >francisfi...@mailup.net
> >___
> >AusNOG mailing list
> >AusNOG@lists.ausnog.net
> >https://lists.ausnog.net/mailman/listinfo/ausnog
> >
>
> ___
> AusNOG mailing list
> AusNOG@lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[DaZZa]

The Optus hamster finally died of old age.

I would suggest your SMS issues would be caused by whoever is issuing
the SMS using Optus - not so much by the Telstra end receiving it.

Anecdotally, Optus enterprise/wholesale appears to be still functional
- at least my link appears to be working fine - and my BGP
advertisements are still being seen overseas - seems to be only NBN
and mobile based services which are busted

D

On Wed, 8 Nov 2023 at 09:27,  wrote:
>
> Morning all,
> Hope the chaos isn't too hard on your work/family.
> I have had trouble with a couple of SMS verifications coming through to me, 
> my Telstra number. Is this related?
>
> Any general banter around the downtime would be fine too - looks like it all 
> began at 4.07am AEDT?
>
> Cheers
>
> --
>
>   francisfi...@mailup.net
> ___
> AusNOG mailing list
> AusNOG@lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog



-- 
veg·e·tar·i·an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[John Edwards]

The 4am Wednesday morning outage start looks suspiciously like a firmware
upgrade window.

I note that Optus devices where I am are showing "SoS" which indicates the
tower is unable to reach the location register, which presumably is on a
private network and indicative of a pretty major fault rather than just IP.

John


On Wed, 8 Nov 2023 at 09:10, DaZZa  wrote:

> The Optus hamster finally died of old age.
>
> I would suggest your SMS issues would be caused by whoever is issuing
> the SMS using Optus - not so much by the Telstra end receiving it.
>
> Anecdotally, Optus enterprise/wholesale appears to be still functional
> - at least my link appears to be working fine - and my BGP
> advertisements are still being seen overseas - seems to be only NBN
> and mobile based services which are busted
>
> D
>
> On Wed, 8 Nov 2023 at 09:27,  wrote:
> >
> > Morning all,
> > Hope the chaos isn't too hard on your work/family.
> > I have had trouble with a couple of SMS verifications coming through to
> me, my Telstra number. Is this related?
> >
> > Any general banter around the downtime would be fine too - looks like it
> all began at 4.07am AEDT?
> >
> > Cheers
> >
> > --
> >
> >   francisfi...@mailup.net
> > ___
> > AusNOG mailing list
> > AusNOG@lists.ausnog.net
> > https://lists.ausnog.net/mailman/listinfo/ausnog
>
>
>
> --
> veg·e·tar·i·an:
> Ancient tribal slang for the village idiot who can't hunt, fish or ride
> ___
> AusNOG mailing list
> AusNOG@lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[DaZZa]

Yeah, I'd be willing to bet that it's a change which wasn't thoroughly
tested before being rolled out, and which had an inadequate backout
plan.

Interestingly, my Optus mobile actually had a valid connection for a
short time - wasn't able to actually DO anything, but was connected to
the OPtus network - but it's now gone to "SOS" mode.

D

On Wed, 8 Nov 2023 at 10:01, John Edwards  wrote:
>
> The 4am Wednesday morning outage start looks suspiciously like a firmware 
> upgrade window.
>
> I note that Optus devices where I am are showing "SoS" which indicates the 
> tower is unable to reach the location register, which presumably is on a 
> private network and indicative of a pretty major fault rather than just IP.
>
> John
>
>
> On Wed, 8 Nov 2023 at 09:10, DaZZa  wrote:
>>
>> The Optus hamster finally died of old age.
>>
>> I would suggest your SMS issues would be caused by whoever is issuing
>> the SMS using Optus - not so much by the Telstra end receiving it.
>>
>> Anecdotally, Optus enterprise/wholesale appears to be still functional
>> - at least my link appears to be working fine - and my BGP
>> advertisements are still being seen overseas - seems to be only NBN
>> and mobile based services which are busted
>>
>> D
>>
>> On Wed, 8 Nov 2023 at 09:27,  wrote:
>> >
>> > Morning all,
>> > Hope the chaos isn't too hard on your work/family.
>> > I have had trouble with a couple of SMS verifications coming through to 
>> > me, my Telstra number. Is this related?
>> >
>> > Any general banter around the downtime would be fine too - looks like it 
>> > all began at 4.07am AEDT?
>> >
>> > Cheers
>> >
>> > --
>> >
>> >   francisfi...@mailup.net
>> > ___
>> > AusNOG mailing list
>> > AusNOG@lists.ausnog.net
>> > https://lists.ausnog.net/mailman/listinfo/ausnog
>>
>>
>>
>> --
>> veg·e·tar·i·an:
>> Ancient tribal slang for the village idiot who can't hunt, fish or ride
>> ___
>> AusNOG mailing list
>> AusNOG@lists.ausnog.net
>> https://lists.ausnog.net/mailman/listinfo/ausnog



-- 
veg·e·tar·i·an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[Ben Buxton]

On Wed, 8 Nov 2023 at 09:29, Joseph Goldman  wrote:

>
> I would also have to imagine the outage is a bad config pushed out -
> surely a hardware failure or fibre cut wouldn't cause such a big outage.
> Bad day to have your name on the last push :/.
>

In a blameless culture, I'l probably consider it a badge of honour to have
my name on the push, but this is Optus so some engineer will be thrown
under the bus.

(In an even better culture though, no human would be manually performing
any push...but again this is Optus...)

BB



>
> -- Original Message --
> From: francisfi...@mailup.net
> To: AusNOG@lists.ausnog.net
> Sent: 8/11/2023 9:26:26 AM
> Subject: [AusNOG] Optus downtime chat + affecting SMS verification to
> Telstra?
>
> >Morning all,
> >Hope the chaos isn't too hard on your work/family.
> >I have had trouble with a couple of SMS verifications coming through to
> me, my Telstra number. Is this related?
> >
> >Any general banter around the downtime would be fine too - looks like it
> all began at 4.07am AEDT?
> >
> >Cheers
> >
> >--
> >
> >francisfi...@mailup.net
> >___
> >AusNOG mailing list
> >AusNOG@lists.ausnog.net
> >https://lists.ausnog.net/mailman/listinfo/ausnog
> >
>
> ___
> AusNOG mailing list
> AusNOG@lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[Bradley Amm]

https://www.abc.net.au/news/2023-11-08/optus-outage-live-blog/103076996

Get Outlook for iOS

From: AusNOG  on behalf of DaZZa 

Sent: Wednesday, November 8, 2023 6:39:22 AM
To: francisfi...@mailup.net 
Cc: AusNOG@lists.ausnog.net 
Subject: Re: [AusNOG] Optus downtime chat + affecting SMS verification to 
Telstra?

The Optus hamster finally died of old age.

I would suggest your SMS issues would be caused by whoever is issuing
the SMS using Optus - not so much by the Telstra end receiving it.

Anecdotally, Optus enterprise/wholesale appears to be still functional
- at least my link appears to be working fine - and my BGP
advertisements are still being seen overseas - seems to be only NBN
and mobile based services which are busted

D

On Wed, 8 Nov 2023 at 09:27,  wrote:
>
> Morning all,
> Hope the chaos isn't too hard on your work/family.
> I have had trouble with a couple of SMS verifications coming through to me, 
> my Telstra number. Is this related?
>
> Any general banter around the downtime would be fine too - looks like it all 
> began at 4.07am AEDT?
>
> Cheers
>
> --
>
>   francisfi...@mailup.net
> ___
> AusNOG mailing list
> AusNOG@lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog



--
veg·e·tar·i·an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[Bradley Amm]

Maybe they were annoyed at glen maxwell getting 200

Get Outlook for iOS

From: AusNOG  on behalf of 
francisfi...@mailup.net 
Sent: Wednesday, November 8, 2023 6:26:26 AM
To: AusNOG@lists.ausnog.net 
Subject: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

Morning all,
Hope the chaos isn't too hard on your work/family.
I have had trouble with a couple of SMS verifications coming through to me, my 
Telstra number. Is this related?

Any general banter around the downtime would be fine too - looks like it all 
began at 4.07am AEDT?

Cheers

--

  francisfi...@mailup.net
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[yahoo]

I happened to be looking at my phone when it happened. It said words to the 
effect of 'the carrier has turned off mobile data for a short time" with a 
little triangle symbol that I've never seen before.I guess it wasn't wrong 
depending on how you define shortSent from my Galaxy
 Original message From: francisfi...@mailup.net Date: 8/11/23  
9:27 am  (GMT+10:00) To: AusNOG@lists.ausnog.net Subject: [AusNOG] Optus 
downtime chat + affecting SMS verification to
Telstra? Morning all,Hope the chaos isn't too hard on your work/family. 
I have had trouble with a couple of SMS verifications coming through to me, my 
Telstra number. Is this related?Any general banter around the downtime would be 
fine too - looks like it all began at 4.07am AEDT?Cheers--     
francisfides@mailup.net___AusNOG 
mailing 
listAusNOG@lists.ausnog.nethttps://lists.ausnog.net/mailman/listinfo/ausnog___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[Robert Hudson]

Gotta say, that interview didn't give me any confidence at all.  It makes
me wonder if the CEO even knows what Optus does as its core business, or
has had any sort of media training on how to communicate with customers and
the public.

"We'd like the media to help by directing people to our messaging" - that
they can't access...

On Wed, 8 Nov 2023 at 10:33, Bradley Amm  wrote:

> https://www.abc.net.au/news/2023-11-08/optus-outage-live-blog/103076996
>
> Get Outlook for iOS 
> --
> *From:* AusNOG  on behalf of DaZZa <
> dazzagi...@gmail.com>
> *Sent:* Wednesday, November 8, 2023 6:39:22 AM
> *To:* francisfi...@mailup.net 
> *Cc:* AusNOG@lists.ausnog.net 
> *Subject:* Re: [AusNOG] Optus downtime chat + affecting SMS verification
> to Telstra?
>
> The Optus hamster finally died of old age.
>
> I would suggest your SMS issues would be caused by whoever is issuing
> the SMS using Optus - not so much by the Telstra end receiving it.
>
> Anecdotally, Optus enterprise/wholesale appears to be still functional
> - at least my link appears to be working fine - and my BGP
> advertisements are still being seen overseas - seems to be only NBN
> and mobile based services which are busted
>
> D
>
> On Wed, 8 Nov 2023 at 09:27,  wrote:
> >
> > Morning all,
> > Hope the chaos isn't too hard on your work/family.
> > I have had trouble with a couple of SMS verifications coming through to
> me, my Telstra number. Is this related?
> >
> > Any general banter around the downtime would be fine too - looks like it
> all began at 4.07am AEDT?
> >
> > Cheers
> >
> > --
> >
> >   francisfi...@mailup.net
> > ___
> > AusNOG mailing list
> > AusNOG@lists.ausnog.net
> > https://lists.ausnog.net/mailman/listinfo/ausnog
>
>
>
> --
> veg·e·tar·i·an:
> Ancient tribal slang for the village idiot who can't hunt, fish or ride
> ___
> AusNOG mailing list
> AusNOG@lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
> ___
> AusNOG mailing list
> AusNOG@lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[DaZZa]

I loved the scramble to avoid saying "We f* up". And the dodge
when asked "How many staff are on-shore and how manby overseas". I
laughed

She's a talking head. I didn't expect any information from her - what
I took out of it is that 7 hours after this started, they *still*
don't know what caused it, or how to fix it.

D

On Wed, 8 Nov 2023 at 10:55, Robert Hudson  wrote:
>
> Gotta say, that interview didn't give me any confidence at all.  It makes me 
> wonder if the CEO even knows what Optus does as its core business, or has had 
> any sort of media training on how to communicate with customers and the 
> public.
>
> "We'd like the media to help by directing people to our messaging" - that 
> they can't access...
>
> On Wed, 8 Nov 2023 at 10:33, Bradley Amm  wrote:
>>
>> https://www.abc.net.au/news/2023-11-08/optus-outage-live-blog/103076996
>>
>> Get Outlook for iOS
>> 
>> From: AusNOG  on behalf of DaZZa 
>> 
>> Sent: Wednesday, November 8, 2023 6:39:22 AM
>> To: francisfi...@mailup.net 
>> Cc: AusNOG@lists.ausnog.net 
>> Subject: Re: [AusNOG] Optus downtime chat + affecting SMS verification to 
>> Telstra?
>>
>> The Optus hamster finally died of old age.
>>
>> I would suggest your SMS issues would be caused by whoever is issuing
>> the SMS using Optus - not so much by the Telstra end receiving it.
>>
>> Anecdotally, Optus enterprise/wholesale appears to be still functional
>> - at least my link appears to be working fine - and my BGP
>> advertisements are still being seen overseas - seems to be only NBN
>> and mobile based services which are busted
>>
>> D
>>
>> On Wed, 8 Nov 2023 at 09:27,  wrote:
>> >
>> > Morning all,
>> > Hope the chaos isn't too hard on your work/family.
>> > I have had trouble with a couple of SMS verifications coming through to 
>> > me, my Telstra number. Is this related?
>> >
>> > Any general banter around the downtime would be fine too - looks like it 
>> > all began at 4.07am AEDT?
>> >
>> > Cheers
>> >
>> > --
>> >
>> >   francisfi...@mailup.net
>> > ___
>> > AusNOG mailing list
>> > AusNOG@lists.ausnog.net
>> > https://lists.ausnog.net/mailman/listinfo/ausnog
>>
>>
>>
>> --
>> veg·e·tar·i·an:
>> Ancient tribal slang for the village idiot who can't hunt, fish or ride
>> ___
>> AusNOG mailing list
>> AusNOG@lists.ausnog.net
>> https://lists.ausnog.net/mailman/listinfo/ausnog
>> ___
>> AusNOG mailing list
>> AusNOG@lists.ausnog.net
>> https://lists.ausnog.net/mailman/listinfo/ausnog



-- 
veg·e·tar·i·an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[Ben Buxton]

On Wed, 8 Nov 2023 at 10:14, DaZZa  wrote:

> Yeah, I'd be willing to bet that it's a change which wasn't thoroughly
> tested before being rolled out, and which had an inadequate backout
> plan.
>

Also, "Our on-site technician is actively prioritising establishing a
console connection.".

I mean come on, it's nearly 2024 and a [major] telco does not have remote
console access? Whilst I'm
looking forward to enthusiastically reading the PM, I'll have to book a
physio appointment in advance due to
neck strain from all the head shaking it'll likely induce.

BB



>
> Interestingly, my Optus mobile actually had a valid connection for a
> short time - wasn't able to actually DO anything, but was connected to
> the OPtus network - but it's now gone to "SOS" mode.
>
> D
>
> On Wed, 8 Nov 2023 at 10:01, John Edwards  wrote:
> >
> > The 4am Wednesday morning outage start looks suspiciously like a
> firmware upgrade window.
> >
> > I note that Optus devices where I am are showing "SoS" which indicates
> the tower is unable to reach the location register, which presumably is on
> a private network and indicative of a pretty major fault rather than just
> IP.
> >
> > John
> >
> >
> > On Wed, 8 Nov 2023 at 09:10, DaZZa  wrote:
> >>
> >> The Optus hamster finally died of old age.
> >>
> >> I would suggest your SMS issues would be caused by whoever is issuing
> >> the SMS using Optus - not so much by the Telstra end receiving it.
> >>
> >> Anecdotally, Optus enterprise/wholesale appears to be still functional
> >> - at least my link appears to be working fine - and my BGP
> >> advertisements are still being seen overseas - seems to be only NBN
> >> and mobile based services which are busted
> >>
> >> D
> >>
> >> On Wed, 8 Nov 2023 at 09:27,  wrote:
> >> >
> >> > Morning all,
> >> > Hope the chaos isn't too hard on your work/family.
> >> > I have had trouble with a couple of SMS verifications coming through
> to me, my Telstra number. Is this related?
> >> >
> >> > Any general banter around the downtime would be fine too - looks like
> it all began at 4.07am AEDT?
> >> >
> >> > Cheers
> >> >
> >> > --
> >> >
> >> >   francisfi...@mailup.net
> >> > ___
> >> > AusNOG mailing list
> >> > AusNOG@lists.ausnog.net
> >> > https://lists.ausnog.net/mailman/listinfo/ausnog
> >>
> >>
> >>
> >> --
> >> veg·e·tar·i·an:
> >> Ancient tribal slang for the village idiot who can't hunt, fish or ride
> >> ___
> >> AusNOG mailing list
> >> AusNOG@lists.ausnog.net
> >> https://lists.ausnog.net/mailman/listinfo/ausnog
>
>
>
> --
> veg·e·tar·i·an:
> Ancient tribal slang for the village idiot who can't hunt, fish or ride
> ___
> AusNOG mailing list
> AusNOG@lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[Nick Jenkin]

Just a few announcements...
https://radar.cloudflare.com/traffic/as4804?dateRange=2d

On Wed, 8 Nov 2023 at 11:06, Ben Buxton  wrote:

>
>
> On Wed, 8 Nov 2023 at 10:14, DaZZa  wrote:
>
>> Yeah, I'd be willing to bet that it's a change which wasn't thoroughly
>> tested before being rolled out, and which had an inadequate backout
>> plan.
>>
>
> Also, "Our on-site technician is actively prioritising establishing a
> console connection.".
>
> I mean come on, it's nearly 2024 and a [major] telco does not have remote
> console access? Whilst I'm
> looking forward to enthusiastically reading the PM, I'll have to book a
> physio appointment in advance due to
> neck strain from all the head shaking it'll likely induce.
>
> BB
>
>
>
>>
>> Interestingly, my Optus mobile actually had a valid connection for a
>> short time - wasn't able to actually DO anything, but was connected to
>> the OPtus network - but it's now gone to "SOS" mode.
>>
>> D
>>
>> On Wed, 8 Nov 2023 at 10:01, John Edwards  wrote:
>> >
>> > The 4am Wednesday morning outage start looks suspiciously like a
>> firmware upgrade window.
>> >
>> > I note that Optus devices where I am are showing "SoS" which indicates
>> the tower is unable to reach the location register, which presumably is on
>> a private network and indicative of a pretty major fault rather than just
>> IP.
>> >
>> > John
>> >
>> >
>> > On Wed, 8 Nov 2023 at 09:10, DaZZa  wrote:
>> >>
>> >> The Optus hamster finally died of old age.
>> >>
>> >> I would suggest your SMS issues would be caused by whoever is issuing
>> >> the SMS using Optus - not so much by the Telstra end receiving it.
>> >>
>> >> Anecdotally, Optus enterprise/wholesale appears to be still functional
>> >> - at least my link appears to be working fine - and my BGP
>> >> advertisements are still being seen overseas - seems to be only NBN
>> >> and mobile based services which are busted
>> >>
>> >> D
>> >>
>> >> On Wed, 8 Nov 2023 at 09:27,  wrote:
>> >> >
>> >> > Morning all,
>> >> > Hope the chaos isn't too hard on your work/family.
>> >> > I have had trouble with a couple of SMS verifications coming through
>> to me, my Telstra number. Is this related?
>> >> >
>> >> > Any general banter around the downtime would be fine too - looks
>> like it all began at 4.07am AEDT?
>> >> >
>> >> > Cheers
>> >> >
>> >> > --
>> >> >
>> >> >   francisfi...@mailup.net
>> >> > ___
>> >> > AusNOG mailing list
>> >> > AusNOG@lists.ausnog.net
>> >> > https://lists.ausnog.net/mailman/listinfo/ausnog
>> >>
>> >>
>> >>
>> >> --
>> >> veg·e·tar·i·an:
>> >> Ancient tribal slang for the village idiot who can't hunt, fish or ride
>> >> ___
>> >> AusNOG mailing list
>> >> AusNOG@lists.ausnog.net
>> >> https://lists.ausnog.net/mailman/listinfo/ausnog
>>
>>
>>
>> --
>> veg·e·tar·i·an:
>> Ancient tribal slang for the village idiot who can't hunt, fish or ride
>> ___
>> AusNOG mailing list
>> AusNOG@lists.ausnog.net
>> https://lists.ausnog.net/mailman/listinfo/ausnog
>>
> ___
> AusNOG mailing list
> AusNOG@lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[Mark Andrews]

I’m wondering if it is leaking over to Vodafone as International Roaming has 
failed for me here in Prague at about the same time.  All I’m getting is “no 
servíce” even after forcing the carrier to Vodafone CZ which I’ve been using 
the last 8 days. 

-- 
Mark Andrews

> On 8 Nov 2023, at 01:10, Nick Jenkin  wrote:
> 
> 
> Just a few announcements... 
> https://radar.cloudflare.com/traffic/as4804?dateRange=2d
> 
>> On Wed, 8 Nov 2023 at 11:06, Ben Buxton  wrote:
>> 
>> 
>>> On Wed, 8 Nov 2023 at 10:14, DaZZa  wrote:
>>> Yeah, I'd be willing to bet that it's a change which wasn't thoroughly
>>> tested before being rolled out, and which had an inadequate backout
>>> plan.
>> 
>> Also, "Our on-site technician is actively prioritising establishing a 
>> console connection.".
>> 
>> I mean come on, it's nearly 2024 and a [major] telco does not have remote 
>> console access? Whilst I'm
>> looking forward to enthusiastically reading the PM, I'll have to book a 
>> physio appointment in advance due to 
>> neck strain from all the head shaking it'll likely induce.
>> 
>> BB
>> 
>>  
>>> 
>>> Interestingly, my Optus mobile actually had a valid connection for a
>>> short time - wasn't able to actually DO anything, but was connected to
>>> the OPtus network - but it's now gone to "SOS" mode.
>>> 
>>> D
>>> 
>>> On Wed, 8 Nov 2023 at 10:01, John Edwards  wrote:
>>> >
>>> > The 4am Wednesday morning outage start looks suspiciously like a firmware 
>>> > upgrade window.
>>> >
>>> > I note that Optus devices where I am are showing "SoS" which indicates 
>>> > the tower is unable to reach the location register, which presumably is 
>>> > on a private network and indicative of a pretty major fault rather than 
>>> > just IP.
>>> >
>>> > John
>>> >
>>> >
>>> > On Wed, 8 Nov 2023 at 09:10, DaZZa  wrote:
>>> >>
>>> >> The Optus hamster finally died of old age.
>>> >>
>>> >> I would suggest your SMS issues would be caused by whoever is issuing
>>> >> the SMS using Optus - not so much by the Telstra end receiving it.
>>> >>
>>> >> Anecdotally, Optus enterprise/wholesale appears to be still functional
>>> >> - at least my link appears to be working fine - and my BGP
>>> >> advertisements are still being seen overseas - seems to be only NBN
>>> >> and mobile based services which are busted
>>> >>
>>> >> D
>>> >>
>>> >> On Wed, 8 Nov 2023 at 09:27,  wrote:
>>> >> >
>>> >> > Morning all,
>>> >> > Hope the chaos isn't too hard on your work/family.
>>> >> > I have had trouble with a couple of SMS verifications coming through 
>>> >> > to me, my Telstra number. Is this related?
>>> >> >
>>> >> > Any general banter around the downtime would be fine too - looks like 
>>> >> > it all began at 4.07am AEDT?
>>> >> >
>>> >> > Cheers
>>> >> >
>>> >> > --
>>> >> >
>>> >> >   francisfi...@mailup.net
>>> >> > ___
>>> >> > AusNOG mailing list
>>> >> > AusNOG@lists.ausnog.net
>>> >> > https://lists.ausnog.net/mailman/listinfo/ausnog
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> veg·e·tar·i·an:
>>> >> Ancient tribal slang for the village idiot who can't hunt, fish or ride
>>> >> ___
>>> >> AusNOG mailing list
>>> >> AusNOG@lists.ausnog.net
>>> >> https://lists.ausnog.net/mailman/listinfo/ausnog
>>> 
>>> 
>>> 
>>> -- 
>>> veg·e·tar·i·an:
>>> Ancient tribal slang for the village idiot who can't hunt, fish or ride
>>> ___
>>> AusNOG mailing list
>>> AusNOG@lists.ausnog.net
>>> https://lists.ausnog.net/mailman/listinfo/ausnog
>> ___
>> AusNOG mailing list
>> AusNOG@lists.ausnog.net
>> https://lists.ausnog.net/mailman/listinfo/ausnog
> ___
> AusNOG mailing list
> AusNOG@lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[Jake Anderson]

Looks like they may getting some services back
https://radar.cloudflare.com/as4804?dateStart=2023-11-07&dateEnd=2023-11-07

On 8/11/2023 9:26 am, francisfi...@mailup.net wrote:

Morning all,
Hope the chaos isn't too hard on your work/family.
I have had trouble with a couple of SMS verifications coming through to me, my 
Telstra number. Is this related?

Any general banter around the downtime would be fine too - looks like it all 
began at 4.07am AEDT?

Cheers



___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[Tim Malone]

Cloudflare is also reporting BGP origin hijack warnings:
https://radar.cloudflare.com/as7474?dateRange=2d
(unsure if related though)


On Wed, 8 Nov 2023 at 11:10, Nick Jenkin  wrote:

> Just a few announcements...
> https://radar.cloudflare.com/traffic/as4804?dateRange=2d
>
> On Wed, 8 Nov 2023 at 11:06, Ben Buxton  wrote:
>
>>
>>
>> On Wed, 8 Nov 2023 at 10:14, DaZZa  wrote:
>>
>>> Yeah, I'd be willing to bet that it's a change which wasn't thoroughly
>>> tested before being rolled out, and which had an inadequate backout
>>> plan.
>>>
>>
>> Also, "Our on-site technician is actively prioritising establishing a
>> console connection.".
>>
>> I mean come on, it's nearly 2024 and a [major] telco does not have remote
>> console access? Whilst I'm
>> looking forward to enthusiastically reading the PM, I'll have to book a
>> physio appointment in advance due to
>> neck strain from all the head shaking it'll likely induce.
>>
>> BB
>>
>>
>>
>>>
>>> Interestingly, my Optus mobile actually had a valid connection for a
>>> short time - wasn't able to actually DO anything, but was connected to
>>> the OPtus network - but it's now gone to "SOS" mode.
>>>
>>> D
>>>
>>> On Wed, 8 Nov 2023 at 10:01, John Edwards  wrote:
>>> >
>>> > The 4am Wednesday morning outage start looks suspiciously like a
>>> firmware upgrade window.
>>> >
>>> > I note that Optus devices where I am are showing "SoS" which indicates
>>> the tower is unable to reach the location register, which presumably is on
>>> a private network and indicative of a pretty major fault rather than just
>>> IP.
>>> >
>>> > John
>>> >
>>> >
>>> > On Wed, 8 Nov 2023 at 09:10, DaZZa  wrote:
>>> >>
>>> >> The Optus hamster finally died of old age.
>>> >>
>>> >> I would suggest your SMS issues would be caused by whoever is issuing
>>> >> the SMS using Optus - not so much by the Telstra end receiving it.
>>> >>
>>> >> Anecdotally, Optus enterprise/wholesale appears to be still functional
>>> >> - at least my link appears to be working fine - and my BGP
>>> >> advertisements are still being seen overseas - seems to be only NBN
>>> >> and mobile based services which are busted
>>> >>
>>> >> D
>>> >>
>>> >> On Wed, 8 Nov 2023 at 09:27,  wrote:
>>> >> >
>>> >> > Morning all,
>>> >> > Hope the chaos isn't too hard on your work/family.
>>> >> > I have had trouble with a couple of SMS verifications coming
>>> through to me, my Telstra number. Is this related?
>>> >> >
>>> >> > Any general banter around the downtime would be fine too - looks
>>> like it all began at 4.07am AEDT?
>>> >> >
>>> >> > Cheers
>>> >> >
>>> >> > --
>>> >> >
>>> >> >   francisfi...@mailup.net
>>> >> > ___
>>> >> > AusNOG mailing list
>>> >> > AusNOG@lists.ausnog.net
>>> >> > https://lists.ausnog.net/mailman/listinfo/ausnog
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> veg·e·tar·i·an:
>>> >> Ancient tribal slang for the village idiot who can't hunt, fish or
>>> ride
>>> >> ___
>>> >> AusNOG mailing list
>>> >> AusNOG@lists.ausnog.net
>>> >> https://lists.ausnog.net/mailman/listinfo/ausnog
>>>
>>>
>>>
>>> --
>>> veg·e·tar·i·an:
>>> Ancient tribal slang for the village idiot who can't hunt, fish or ride
>>> ___
>>> AusNOG mailing list
>>> AusNOG@lists.ausnog.net
>>> https://lists.ausnog.net/mailman/listinfo/ausnog
>>>
>> ___
>> AusNOG mailing list
>> AusNOG@lists.ausnog.net
>> https://lists.ausnog.net/mailman/listinfo/ausnog
>>
> ___
> AusNOG mailing list
> AusNOG@lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[Christopher Hawker]

They weren't appearing earlier so I'd say it's new. Smells like BGP has shat 
itself.

- ​CH

From: AusNOG  on behalf of Tim Malone 

Sent: Wednesday, November 8, 2023 12:16 PM
To: Nick Jenkin 
Cc: ausnog@lists.ausnog.net 
Subject: Re: [AusNOG] Optus downtime chat + affecting SMS verification to 
Telstra?

Cloudflare is also reporting BGP origin hijack warnings:
https://radar.cloudflare.com/as7474?dateRange=2d
(unsure if related though)


On Wed, 8 Nov 2023 at 11:10, Nick Jenkin 
mailto:njen...@gmail.com>> wrote:
Just a few announcements... 
https://radar.cloudflare.com/traffic/as4804?dateRange=2d

On Wed, 8 Nov 2023 at 11:06, Ben Buxton 
mailto:bb.aus...@bb.cactii.net>> wrote:


On Wed, 8 Nov 2023 at 10:14, DaZZa 
mailto:dazzagi...@gmail.com>> wrote:
Yeah, I'd be willing to bet that it's a change which wasn't thoroughly
tested before being rolled out, and which had an inadequate backout
plan.

Also, "Our on-site technician is actively prioritising establishing a console 
connection.".

I mean come on, it's nearly 2024 and a [major] telco does not have remote 
console access? Whilst I'm
looking forward to enthusiastically reading the PM, I'll have to book a physio 
appointment in advance due to
neck strain from all the head shaking it'll likely induce.

BB



Interestingly, my Optus mobile actually had a valid connection for a
short time - wasn't able to actually DO anything, but was connected to
the OPtus network - but it's now gone to "SOS" mode.

D

On Wed, 8 Nov 2023 at 10:01, John Edwards 
mailto:jaedwa...@gmail.com>> wrote:
>
> The 4am Wednesday morning outage start looks suspiciously like a firmware 
> upgrade window.
>
> I note that Optus devices where I am are showing "SoS" which indicates the 
> tower is unable to reach the location register, which presumably is on a 
> private network and indicative of a pretty major fault rather than just IP.
>
> John
>
>
> On Wed, 8 Nov 2023 at 09:10, DaZZa 
> mailto:dazzagi...@gmail.com>> wrote:
>>
>> The Optus hamster finally died of old age.
>>
>> I would suggest your SMS issues would be caused by whoever is issuing
>> the SMS using Optus - not so much by the Telstra end receiving it.
>>
>> Anecdotally, Optus enterprise/wholesale appears to be still functional
>> - at least my link appears to be working fine - and my BGP
>> advertisements are still being seen overseas - seems to be only NBN
>> and mobile based services which are busted
>>
>> D
>>
>> On Wed, 8 Nov 2023 at 09:27, 
>> mailto:francisfi...@mailup.net>> wrote:
>> >
>> > Morning all,
>> > Hope the chaos isn't too hard on your work/family.
>> > I have had trouble with a couple of SMS verifications coming through to 
>> > me, my Telstra number. Is this related?
>> >
>> > Any general banter around the downtime would be fine too - looks like it 
>> > all began at 4.07am AEDT?
>> >
>> > Cheers
>> >
>> > --
>> >
>> >   francisfi...@mailup.net
>> > ___
>> > AusNOG mailing list
>> > AusNOG@lists.ausnog.net
>> > https://lists.ausnog.net/mailman/listinfo/ausnog
>>
>>
>>
>> --
>> veg·e·tar·i·an:
>> Ancient tribal slang for the village idiot who can't hunt, fish or ride
>> ___
>> AusNOG mailing list
>> AusNOG@lists.ausnog.net
>> https://lists.ausnog.net/mailman/listinfo/ausnog



--
veg·e·tar·i·an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[John Edwards]

Those who believe it's a BGP issue might want to review
https://www.cidr-report.org/cgi-bin/plota?file=%2fvar%2fdata%2fbgp%2fas2.0%2fbgp%2dactive%2etxt&descr=Active%20BGP%20entries%20%28FIB%29&ylabel=Active%20BGP%20entries%20%28FIB%29&with=step

This shows a global table getting dangerously close to a 1024K boundary
found in some router FIBs.

John


On Wed, 8 Nov 2023 at 11:57, Christopher Hawker 
wrote:

> They weren't appearing earlier so I'd say it's new. Smells like BGP has
> shat itself.
>
> - ​CH
> --
> *From:* AusNOG  on behalf of Tim Malone <
> tdmal...@gmail.com>
> *Sent:* Wednesday, November 8, 2023 12:16 PM
> *To:* Nick Jenkin 
> *Cc:* ausnog@lists.ausnog.net 
> *Subject:* Re: [AusNOG] Optus downtime chat + affecting SMS verification
> to Telstra?
>
> Cloudflare is also reporting BGP origin hijack warnings:
> https://radar.cloudflare.com/as7474?dateRange=2d
> (unsure if related though)
>
>
> On Wed, 8 Nov 2023 at 11:10, Nick Jenkin  wrote:
>
> Just a few announcements...
> https://radar.cloudflare.com/traffic/as4804?dateRange=2d
>
> On Wed, 8 Nov 2023 at 11:06, Ben Buxton  wrote:
>
>
>
> On Wed, 8 Nov 2023 at 10:14, DaZZa  wrote:
>
> Yeah, I'd be willing to bet that it's a change which wasn't thoroughly
> tested before being rolled out, and which had an inadequate backout
> plan.
>
>
> Also, "Our on-site technician is actively prioritising establishing a
> console connection.".
>
> I mean come on, it's nearly 2024 and a [major] telco does not have remote
> console access? Whilst I'm
> looking forward to enthusiastically reading the PM, I'll have to book a
> physio appointment in advance due to
> neck strain from all the head shaking it'll likely induce.
>
> BB
>
>
>
>
> Interestingly, my Optus mobile actually had a valid connection for a
> short time - wasn't able to actually DO anything, but was connected to
> the OPtus network - but it's now gone to "SOS" mode.
>
> D
>
> On Wed, 8 Nov 2023 at 10:01, John Edwards  wrote:
> >
> > The 4am Wednesday morning outage start looks suspiciously like a
> firmware upgrade window.
> >
> > I note that Optus devices where I am are showing "SoS" which indicates
> the tower is unable to reach the location register, which presumably is on
> a private network and indicative of a pretty major fault rather than just
> IP.
> >
> > John
> >
> >
> > On Wed, 8 Nov 2023 at 09:10, DaZZa  wrote:
> >>
> >> The Optus hamster finally died of old age.
> >>
> >> I would suggest your SMS issues would be caused by whoever is issuing
> >> the SMS using Optus - not so much by the Telstra end receiving it.
> >>
> >> Anecdotally, Optus enterprise/wholesale appears to be still functional
> >> - at least my link appears to be working fine - and my BGP
> >> advertisements are still being seen overseas - seems to be only NBN
> >> and mobile based services which are busted
> >>
> >> D
> >>
> >> On Wed, 8 Nov 2023 at 09:27,  wrote:
> >> >
> >> > Morning all,
> >> > Hope the chaos isn't too hard on your work/family.
> >> > I have had trouble with a couple of SMS verifications coming through
> to me, my Telstra number. Is this related?
> >> >
> >> > Any general banter around the downtime would be fine too - looks like
> it all began at 4.07am AEDT?
> >> >
> >> > Cheers
> >> >
> >> > --
> >> >
> >> >   francisfi...@mailup.net
> >> > ___
> >> > AusNOG mailing list
> >> > AusNOG@lists.ausnog.net
> >> > https://lists.ausnog.net/mailman/listinfo/ausnog
> >>
> >>
> >>
> >> --
> >> veg·e·tar·i·an:
> >> Ancient tribal slang for the village idiot who can't hunt, fish or ride
> >> ___
> >> AusNOG mailing list
> >> AusNOG@lists.ausnog.net
> >> https://lists.ausnog.net/mailman/listinfo/ausnog
>
>
>
> --
> veg·e·tar·i·an:
> Ancient tribal slang for the village idiot who can't hunt, fish or ride
> ___
> AusNOG mailing list
> AusNOG@lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
> ___
> AusNOG mailing list
> AusNOG@lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
> ___
> AusNOG mailing list
> AusNOG@lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
> ___
> AusNOG mailing list
> AusNOG@lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[DaZZa]

The ABC's rolling coverage reports "Some services across fixed and
mobile are now gradually being restored.", but it may take a "few
hours" for everything to recover.

Now we just get to wait for the report on the outage. it oughta be a doozy!

D

On Wed, 8 Nov 2023 at 12:15, Jake Anderson  wrote:
>
> Looks like they may getting some services back
> https://radar.cloudflare.com/as4804?dateStart=2023-11-07&dateEnd=2023-11-07
>
> On 8/11/2023 9:26 am, francisfi...@mailup.net wrote:
> > Morning all,
> > Hope the chaos isn't too hard on your work/family.
> > I have had trouble with a couple of SMS verifications coming through to me, 
> > my Telstra number. Is this related?
> >
> > Any general banter around the downtime would be fine too - looks like it 
> > all began at 4.07am AEDT?
> >
> > Cheers
> >
>
> ___
> AusNOG mailing list
> AusNOG@lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog



-- 
veg·e·tar·i·an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[Jonathon Musters]

Hey All,

This sounds much like what happened in Canada with Rogers –
https://en.wikipedia.org/wiki/2022_Rogers_Communications_outage

The same words seem to be coming from the top 😊



From: AusNOG  on behalf of DaZZa 

Date: Wednesday, November 8, 2023 at 1:10 PM
To: Jake Anderson 
Cc: ausnog@lists.ausnog.net 
Subject: Re: [AusNOG] Optus downtime chat + affecting SMS verification to 
Telstra?
The ABC's rolling coverage reports "Some services across fixed and
mobile are now gradually being restored.", but it may take a "few
hours" for everything to recover.

Now we just get to wait for the report on the outage. it oughta be a doozy!

D

On Wed, 8 Nov 2023 at 12:15, Jake Anderson  wrote:
>
> Looks like they may getting some services back
> https://radar.cloudflare.com/as4804?dateStart=2023-11-07&dateEnd=2023-11-07
>
> On 8/11/2023 9:26 am, francisfi...@mailup.net wrote:
> > Morning all,
> > Hope the chaos isn't too hard on your work/family.
> > I have had trouble with a couple of SMS verifications coming through to me, 
> > my Telstra number. Is this related?
> >
> > Any general banter around the downtime would be fine too - looks like it 
> > all began at 4.07am AEDT?
> >
> > Cheers
> >
>
> ___
> AusNOG mailing list
> AusNOG@lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog



--
veg·e·tar·i·an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog