Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[DaZZa]

I have all three you're asking about.

But I'm very small potatoes compared to most of the members of this
list, and my required remote footprint is correspondingly small, so
it's easy to maintain.

D

On Thu, 9 Nov 2023 at 06:18, Phillip Grasso  wrote:
>>
>> I mean come on, it's nearly 2024 and a [major] telco does not have remote 
>> console access?
>
>
> If we send a poll out to this community, how many would be able to genuinely 
> honestly answer:
>
> Do you have a console or appropriate control plane access into all your 
> critical infrastructure?
> Do you have independant out of band that does not share any infrastructure 
> with your current system(s) - with exemption for physical location and power.
> Do you have the ability to remote power control your devices?
>
> We know from the facebook outage in 2021 that they probably didn't have the 
> above, so its not entirely uncommon for folks to have *proper independant* 
> console and remote access.
>
>
> I empathize with the Optus team and their customers who have been negatively 
> impacted by this incident. I sincerely hope that some positive outcomes can 
> emerge from this situation, including:
>
> - Attention to critical infrastructure resilience
> - BGP clue increases
> - Incident management improves
> (I'm sure there's more).
>
> Network is a black box to most people and I think a large chunk of Australia 
> now knows what it feels like to not have it.
>
>
> On Wed, 8 Nov 2023 at 11:06, Ben Buxton  wrote:
>>
>>
>>
>> On Wed, 8 Nov 2023 at 10:14, DaZZa  wrote:
>>>
>>> Yeah, I'd be willing to bet that it's a change which wasn't thoroughly
>>> tested before being rolled out, and which had an inadequate backout
>>> plan.
>>
>>
>> Also, "Our on-site technician is actively prioritising establishing a 
>> console connection.".
>>
>> I mean come on, it's nearly 2024 and a [major] telco does not have remote 
>> console access? Whilst I'm
>> looking forward to enthusiastically reading the PM, I'll have to book a 
>> physio appointment in advance due to
>> neck strain from all the head shaking it'll likely induce.
>>
>> BB
>>
>>
>>>
>>>
>>> Interestingly, my Optus mobile actually had a valid connection for a
>>> short time - wasn't able to actually DO anything, but was connected to
>>> the OPtus network - but it's now gone to "SOS" mode.
>>>
>>> D
>>>
>>> On Wed, 8 Nov 2023 at 10:01, John Edwards  wrote:
>>> >
>>> > The 4am Wednesday morning outage start looks suspiciously like a firmware 
>>> > upgrade window.
>>> >
>>> > I note that Optus devices where I am are showing "SoS" which indicates 
>>> > the tower is unable to reach the location register, which presumably is 
>>> > on a private network and indicative of a pretty major fault rather than 
>>> > just IP.
>>> >
>>> > John
>>> >
>>> >
>>> > On Wed, 8 Nov 2023 at 09:10, DaZZa  wrote:
>>> >>
>>> >> The Optus hamster finally died of old age.
>>> >>
>>> >> I would suggest your SMS issues would be caused by whoever is issuing
>>> >> the SMS using Optus - not so much by the Telstra end receiving it.
>>> >>
>>> >> Anecdotally, Optus enterprise/wholesale appears to be still functional
>>> >> - at least my link appears to be working fine - and my BGP
>>> >> advertisements are still being seen overseas - seems to be only NBN
>>> >> and mobile based services which are busted
>>> >>
>>> >> D
>>> >>
>>> >> On Wed, 8 Nov 2023 at 09:27,  wrote:
>>> >> >
>>> >> > Morning all,
>>> >> > Hope the chaos isn't too hard on your work/family.
>>> >> > I have had trouble with a couple of SMS verifications coming through 
>>> >> > to me, my Telstra number. Is this related?
>>> >> >
>>> >> > Any general banter around the downtime would be fine too - looks like 
>>> >> > it all began at 4.07am AEDT?
>>> >> >
>>> >> > Cheers
>>> >> >
>>> >> > --
>>> >> >
>>> >> >   francisfi...@mailup.net
>>> >> > ___
>>> >> > AusNOG mailing list
>>> >> > AusNOG@lists.ausnog.net
>>> >> > https://lists.ausnog.net/mailman/listinfo/ausnog
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> veg·e·tar·i·an:
>>> >> Ancient tribal slang for the village idiot who can't hunt, fish or ride
>>> >> ___
>>> >> AusNOG mailing list
>>> >> AusNOG@lists.ausnog.net
>>> >> https://lists.ausnog.net/mailman/listinfo/ausnog
>>>
>>>
>>>
>>> --
>>> veg·e·tar·i·an:
>>> Ancient tribal slang for the village idiot who can't hunt, fish or ride
>>> ___
>>> AusNOG mailing list
>>> AusNOG@lists.ausnog.net
>>> https://lists.ausnog.net/mailman/listinfo/ausnog
>>
>> ___
>> AusNOG mailing list
>> AusNOG@lists.ausnog.net
>> https://lists.ausnog.net/mailman/listinfo/ausnog



-- 
veg·e·tar·i·an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[Phillip Grasso]

>
> I mean come on, it's nearly 2024 and a [major] telco does not have remote
> console access?
>

If we send a poll out to this community, how many would be able to
genuinely honestly answer:

Do you have a console or appropriate control plane access into all your
critical infrastructure?
Do you have independant out of band that does not share any infrastructure
with your current system(s) - with exemption for physical location and
power.
Do you have the ability to remote power control your devices?

We know from the facebook outage in 2021 that they probably didn't have the
above, so its not entirely uncommon for folks to have *proper independant*
console and remote access.


I empathize with the Optus team and their customers who have been
negatively impacted by this incident. I sincerely hope that some positive
outcomes can emerge from this situation, including:

- Attention to critical infrastructure resilience
- BGP clue increases
- Incident management improves
(I'm sure there's more).

Network is a black box to most people and I think a large chunk of
Australia now knows what it feels like to not have it.


On Wed, 8 Nov 2023 at 11:06, Ben Buxton  wrote:

>
>
> On Wed, 8 Nov 2023 at 10:14, DaZZa  wrote:
>
>> Yeah, I'd be willing to bet that it's a change which wasn't thoroughly
>> tested before being rolled out, and which had an inadequate backout
>> plan.
>>
>
> Also, "Our on-site technician is actively prioritising establishing a
> console connection.".
>
> I mean come on, it's nearly 2024 and a [major] telco does not have remote
> console access? Whilst I'm
> looking forward to enthusiastically reading the PM, I'll have to book a
> physio appointment in advance due to
> neck strain from all the head shaking it'll likely induce.
>
> BB
>
>
>
>>
>> Interestingly, my Optus mobile actually had a valid connection for a
>> short time - wasn't able to actually DO anything, but was connected to
>> the OPtus network - but it's now gone to "SOS" mode.
>>
>> D
>>
>> On Wed, 8 Nov 2023 at 10:01, John Edwards  wrote:
>> >
>> > The 4am Wednesday morning outage start looks suspiciously like a
>> firmware upgrade window.
>> >
>> > I note that Optus devices where I am are showing "SoS" which indicates
>> the tower is unable to reach the location register, which presumably is on
>> a private network and indicative of a pretty major fault rather than just
>> IP.
>> >
>> > John
>> >
>> >
>> > On Wed, 8 Nov 2023 at 09:10, DaZZa  wrote:
>> >>
>> >> The Optus hamster finally died of old age.
>> >>
>> >> I would suggest your SMS issues would be caused by whoever is issuing
>> >> the SMS using Optus - not so much by the Telstra end receiving it.
>> >>
>> >> Anecdotally, Optus enterprise/wholesale appears to be still functional
>> >> - at least my link appears to be working fine - and my BGP
>> >> advertisements are still being seen overseas - seems to be only NBN
>> >> and mobile based services which are busted
>> >>
>> >> D
>> >>
>> >> On Wed, 8 Nov 2023 at 09:27,  wrote:
>> >> >
>> >> > Morning all,
>> >> > Hope the chaos isn't too hard on your work/family.
>> >> > I have had trouble with a couple of SMS verifications coming through
>> to me, my Telstra number. Is this related?
>> >> >
>> >> > Any general banter around the downtime would be fine too - looks
>> like it all began at 4.07am AEDT?
>> >> >
>> >> > Cheers
>> >> >
>> >> > --
>> >> >
>> >> >   francisfi...@mailup.net
>> >> > ___
>> >> > AusNOG mailing list
>> >> > AusNOG@lists.ausnog.net
>> >> > https://lists.ausnog.net/mailman/listinfo/ausnog
>> >>
>> >>
>> >>
>> >> --
>> >> veg·e·tar·i·an:
>> >> Ancient tribal slang for the village idiot who can't hunt, fish or ride
>> >> ___
>> >> AusNOG mailing list
>> >> AusNOG@lists.ausnog.net
>> >> https://lists.ausnog.net/mailman/listinfo/ausnog
>>
>>
>>
>> --
>> veg·e·tar·i·an:
>> Ancient tribal slang for the village idiot who can't hunt, fish or ride
>> ___
>> AusNOG mailing list
>> AusNOG@lists.ausnog.net
>> https://lists.ausnog.net/mailman/listinfo/ausnog
>>
> ___
> AusNOG mailing list
> AusNOG@lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[Bradley Amm]

Maybe they had to fly them in from SingaporeSent from my iPadOn 8 Nov 2023, at 7:14 pm, Nathan Brookfield  wrote:




My money is the on-call aren’t in Australia and couldn’t access the network or OOB because it was all on net haha!



On 8 Nov 2023, at 21:03, Damian Guppy  wrote:




Any money on all the on call engineers having Optus mobiles and nbn so couldn’t be contacted?


—Damian


On Wed, Nov 8, 2023 at 6:29 PM Tom Storey  wrote:


If their OOB phone lines or mobile broadband services were on-net they might as well not have had any to begin with. At a guess they probably were otherwise they wouldn't have had to send someone to physically stand in front of a device.


On Wed, 8 Nov 2023, 00:06 Ben Buxton,  wrote:







On Wed, 8 Nov 2023 at 10:14, DaZZa  wrote:


Yeah, I'd be willing to bet that it's a change which wasn't thoroughly
tested before being rolled out, and which had an inadequate backout
plan.



Also, "Our on-site technician is actively prioritising establishing a console connection.".


I mean come on, it's nearly 2024 and a [major] telco does not have remote console access? Whilst I'm
looking forward to enthusiastically reading the PM, I'll have to book a physio appointment in advance due to 
neck strain from all the head shaking it'll likely induce.


BB


 


Interestingly, my Optus mobile actually had a valid connection for a
short time - wasn't able to actually DO anything, but was connected to
the OPtus network - but it's now gone to "SOS" mode.

D

On Wed, 8 Nov 2023 at 10:01, John Edwards  wrote:
>
> The 4am Wednesday morning outage start looks suspiciously like a firmware upgrade window.
>
> I note that Optus devices where I am are showing "SoS" which indicates the tower is unable to reach the location register, which presumably is on a private network and indicative of a pretty major fault rather than just IP.
>
> John
>
>
> On Wed, 8 Nov 2023 at 09:10, DaZZa  wrote:
>>
>> The Optus hamster finally died of old age.
>>
>> I would suggest your SMS issues would be caused by whoever is issuing
>> the SMS using Optus - not so much by the Telstra end receiving it.
>>
>> Anecdotally, Optus enterprise/wholesale appears to be still functional
>> - at least my link appears to be working fine - and my BGP
>> advertisements are still being seen overseas - seems to be only NBN
>> and mobile based services which are busted
>>
>> D
>>
>> On Wed, 8 Nov 2023 at 09:27,  wrote:
>> >
>> > Morning all,
>> > Hope the chaos isn't too hard on your work/family.
>> > I have had trouble with a couple of SMS verifications coming through to me, my Telstra number. Is this related?
>> >
>> > Any general banter around the downtime would be fine too - looks like it all began at 4.07am AEDT?
>> >
>> > Cheers
>> >
>> > --
>> >
>> >   francisfi...@mailup.net
>> > ___
>> > AusNOG mailing list
>> > AusNOG@lists.ausnog.net
>> > 
https://lists.ausnog.net/mailman/listinfo/ausnog
>>
>>
>>
>> --
>> veg·e·tar·i·an:
>> Ancient tribal slang for the village idiot who can't hunt, fish or ride
>> ___
>> AusNOG mailing list
>> AusNOG@lists.ausnog.net
>> 
https://lists.ausnog.net/mailman/listinfo/ausnog



-- 
veg·e·tar·i·an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog



___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog


___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog



___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog





___AusNOG mailing listAusNOG@lists.ausnog.nethttps://lists.ausnog.net/mailman/listinfo/ausnog___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[Nathan Brookfield]

My money is the on-call aren’t in Australia and couldn’t access the network or 
OOB because it was all on net haha!

On 8 Nov 2023, at 21:03, Damian Guppy  wrote:


Any money on all the on call engineers having Optus mobiles and nbn so couldn’t 
be contacted?

—Damian

On Wed, Nov 8, 2023 at 6:29 PM Tom Storey 
mailto:t...@snnap.net>> wrote:
If their OOB phone lines or mobile broadband services were on-net they might as 
well not have had any to begin with. At a guess they probably were otherwise 
they wouldn't have had to send someone to physically stand in front of a device.

On Wed, 8 Nov 2023, 00:06 Ben Buxton, 
mailto:bb.aus...@bb.cactii.net>> wrote:


On Wed, 8 Nov 2023 at 10:14, DaZZa 
mailto:dazzagi...@gmail.com>> wrote:
Yeah, I'd be willing to bet that it's a change which wasn't thoroughly
tested before being rolled out, and which had an inadequate backout
plan.

Also, "Our on-site technician is actively prioritising establishing a console 
connection.".

I mean come on, it's nearly 2024 and a [major] telco does not have remote 
console access? Whilst I'm
looking forward to enthusiastically reading the PM, I'll have to book a physio 
appointment in advance due to
neck strain from all the head shaking it'll likely induce.

BB



Interestingly, my Optus mobile actually had a valid connection for a
short time - wasn't able to actually DO anything, but was connected to
the OPtus network - but it's now gone to "SOS" mode.

D

On Wed, 8 Nov 2023 at 10:01, John Edwards 
mailto:jaedwa...@gmail.com>> wrote:
>
> The 4am Wednesday morning outage start looks suspiciously like a firmware 
> upgrade window.
>
> I note that Optus devices where I am are showing "SoS" which indicates the 
> tower is unable to reach the location register, which presumably is on a 
> private network and indicative of a pretty major fault rather than just IP.
>
> John
>
>
> On Wed, 8 Nov 2023 at 09:10, DaZZa 
> mailto:dazzagi...@gmail.com>> wrote:
>>
>> The Optus hamster finally died of old age.
>>
>> I would suggest your SMS issues would be caused by whoever is issuing
>> the SMS using Optus - not so much by the Telstra end receiving it.
>>
>> Anecdotally, Optus enterprise/wholesale appears to be still functional
>> - at least my link appears to be working fine - and my BGP
>> advertisements are still being seen overseas - seems to be only NBN
>> and mobile based services which are busted
>>
>> D
>>
>> On Wed, 8 Nov 2023 at 09:27, 
>> mailto:francisfi...@mailup.net>> wrote:
>> >
>> > Morning all,
>> > Hope the chaos isn't too hard on your work/family.
>> > I have had trouble with a couple of SMS verifications coming through to 
>> > me, my Telstra number. Is this related?
>> >
>> > Any general banter around the downtime would be fine too - looks like it 
>> > all began at 4.07am AEDT?
>> >
>> > Cheers
>> >
>> > --
>> >
>> >   francisfi...@mailup.net
>> > ___
>> > AusNOG mailing list
>> > AusNOG@lists.ausnog.net
>> > https://lists.ausnog.net/mailman/listinfo/ausnog
>>
>>
>>
>> --
>> veg·e·tar·i·an:
>> Ancient tribal slang for the village idiot who can't hunt, fish or ride
>> ___
>> AusNOG mailing list
>> AusNOG@lists.ausnog.net
>> https://lists.ausnog.net/mailman/listinfo/ausnog



--
veg·e·tar·i·an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[Damian Guppy]

Any money on all the on call engineers having Optus mobiles and nbn so
couldn’t be contacted?

—Damian

On Wed, Nov 8, 2023 at 6:29 PM Tom Storey  wrote:

> If their OOB phone lines or mobile broadband services were on-net they
> might as well not have had any to begin with. At a guess they probably were
> otherwise they wouldn't have had to send someone to physically stand in
> front of a device.
>
> On Wed, 8 Nov 2023, 00:06 Ben Buxton,  wrote:
>
>>
>>
>> On Wed, 8 Nov 2023 at 10:14, DaZZa  wrote:
>>
>>> Yeah, I'd be willing to bet that it's a change which wasn't thoroughly
>>> tested before being rolled out, and which had an inadequate backout
>>> plan.
>>>
>>
>> Also, "Our on-site technician is actively prioritising establishing a
>> console connection.".
>>
>> I mean come on, it's nearly 2024 and a [major] telco does not have remote
>> console access? Whilst I'm
>> looking forward to enthusiastically reading the PM, I'll have to book a
>> physio appointment in advance due to
>> neck strain from all the head shaking it'll likely induce.
>>
>> BB
>>
>>
>>
>>>
>>> Interestingly, my Optus mobile actually had a valid connection for a
>>> short time - wasn't able to actually DO anything, but was connected to
>>> the OPtus network - but it's now gone to "SOS" mode.
>>>
>>> D
>>>
>>> On Wed, 8 Nov 2023 at 10:01, John Edwards  wrote:
>>> >
>>> > The 4am Wednesday morning outage start looks suspiciously like a
>>> firmware upgrade window.
>>> >
>>> > I note that Optus devices where I am are showing "SoS" which indicates
>>> the tower is unable to reach the location register, which presumably is on
>>> a private network and indicative of a pretty major fault rather than just
>>> IP.
>>> >
>>> > John
>>> >
>>> >
>>> > On Wed, 8 Nov 2023 at 09:10, DaZZa  wrote:
>>> >>
>>> >> The Optus hamster finally died of old age.
>>> >>
>>> >> I would suggest your SMS issues would be caused by whoever is issuing
>>> >> the SMS using Optus - not so much by the Telstra end receiving it.
>>> >>
>>> >> Anecdotally, Optus enterprise/wholesale appears to be still functional
>>> >> - at least my link appears to be working fine - and my BGP
>>> >> advertisements are still being seen overseas - seems to be only NBN
>>> >> and mobile based services which are busted
>>> >>
>>> >> D
>>> >>
>>> >> On Wed, 8 Nov 2023 at 09:27,  wrote:
>>> >> >
>>> >> > Morning all,
>>> >> > Hope the chaos isn't too hard on your work/family.
>>> >> > I have had trouble with a couple of SMS verifications coming
>>> through to me, my Telstra number. Is this related?
>>> >> >
>>> >> > Any general banter around the downtime would be fine too - looks
>>> like it all began at 4.07am AEDT?
>>> >> >
>>> >> > Cheers
>>> >> >
>>> >> > --
>>> >> >
>>> >> >   francisfi...@mailup.net
>>> >> > ___
>>> >> > AusNOG mailing list
>>> >> > AusNOG@lists.ausnog.net
>>> >> > https://lists.ausnog.net/mailman/listinfo/ausnog
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> veg·e·tar·i·an:
>>> >> Ancient tribal slang for the village idiot who can't hunt, fish or
>>> ride
>>> >> ___
>>> >> AusNOG mailing list
>>> >> AusNOG@lists.ausnog.net
>>> >> https://lists.ausnog.net/mailman/listinfo/ausnog
>>>
>>>
>>>
>>> --
>>> veg·e·tar·i·an:
>>> Ancient tribal slang for the village idiot who can't hunt, fish or ride
>>> ___
>>> AusNOG mailing list
>>> AusNOG@lists.ausnog.net
>>> https://lists.ausnog.net/mailman/listinfo/ausnog
>>>
>> ___
>> AusNOG mailing list
>> AusNOG@lists.ausnog.net
>> https://lists.ausnog.net/mailman/listinfo/ausnog
>>
> ___
> AusNOG mailing list
> AusNOG@lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[Tom Storey]

If their OOB phone lines or mobile broadband services were on-net they
might as well not have had any to begin with. At a guess they probably were
otherwise they wouldn't have had to send someone to physically stand in
front of a device.

On Wed, 8 Nov 2023, 00:06 Ben Buxton,  wrote:

>
>
> On Wed, 8 Nov 2023 at 10:14, DaZZa  wrote:
>
>> Yeah, I'd be willing to bet that it's a change which wasn't thoroughly
>> tested before being rolled out, and which had an inadequate backout
>> plan.
>>
>
> Also, "Our on-site technician is actively prioritising establishing a
> console connection.".
>
> I mean come on, it's nearly 2024 and a [major] telco does not have remote
> console access? Whilst I'm
> looking forward to enthusiastically reading the PM, I'll have to book a
> physio appointment in advance due to
> neck strain from all the head shaking it'll likely induce.
>
> BB
>
>
>
>>
>> Interestingly, my Optus mobile actually had a valid connection for a
>> short time - wasn't able to actually DO anything, but was connected to
>> the OPtus network - but it's now gone to "SOS" mode.
>>
>> D
>>
>> On Wed, 8 Nov 2023 at 10:01, John Edwards  wrote:
>> >
>> > The 4am Wednesday morning outage start looks suspiciously like a
>> firmware upgrade window.
>> >
>> > I note that Optus devices where I am are showing "SoS" which indicates
>> the tower is unable to reach the location register, which presumably is on
>> a private network and indicative of a pretty major fault rather than just
>> IP.
>> >
>> > John
>> >
>> >
>> > On Wed, 8 Nov 2023 at 09:10, DaZZa  wrote:
>> >>
>> >> The Optus hamster finally died of old age.
>> >>
>> >> I would suggest your SMS issues would be caused by whoever is issuing
>> >> the SMS using Optus - not so much by the Telstra end receiving it.
>> >>
>> >> Anecdotally, Optus enterprise/wholesale appears to be still functional
>> >> - at least my link appears to be working fine - and my BGP
>> >> advertisements are still being seen overseas - seems to be only NBN
>> >> and mobile based services which are busted
>> >>
>> >> D
>> >>
>> >> On Wed, 8 Nov 2023 at 09:27,  wrote:
>> >> >
>> >> > Morning all,
>> >> > Hope the chaos isn't too hard on your work/family.
>> >> > I have had trouble with a couple of SMS verifications coming through
>> to me, my Telstra number. Is this related?
>> >> >
>> >> > Any general banter around the downtime would be fine too - looks
>> like it all began at 4.07am AEDT?
>> >> >
>> >> > Cheers
>> >> >
>> >> > --
>> >> >
>> >> >   francisfi...@mailup.net
>> >> > ___
>> >> > AusNOG mailing list
>> >> > AusNOG@lists.ausnog.net
>> >> > https://lists.ausnog.net/mailman/listinfo/ausnog
>> >>
>> >>
>> >>
>> >> --
>> >> veg·e·tar·i·an:
>> >> Ancient tribal slang for the village idiot who can't hunt, fish or ride
>> >> ___
>> >> AusNOG mailing list
>> >> AusNOG@lists.ausnog.net
>> >> https://lists.ausnog.net/mailman/listinfo/ausnog
>>
>>
>>
>> --
>> veg·e·tar·i·an:
>> Ancient tribal slang for the village idiot who can't hunt, fish or ride
>> ___
>> AusNOG mailing list
>> AusNOG@lists.ausnog.net
>> https://lists.ausnog.net/mailman/listinfo/ausnog
>>
> ___
> AusNOG mailing list
> AusNOG@lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[Steven Waite]

I am sure the outage report would be good reading. Was hearing a lot about 
overloaded route reflectors.

Thanks Steve

From: AusNOG  On Behalf Of Jonathon Musters
Sent: Wednesday, November 8, 2023 3:59 PM
To: ausnog@lists.ausnog.net
Cc: ausnog@lists.ausnog.net
Subject: Re: [AusNOG] Optus downtime chat + affecting SMS verification to 
Telstra?

Hey All,

This sounds much like what happened in Canada with Rogers –
https://en.wikipedia.org/wiki/2022_Rogers_Communications_outage

The same words seem to be coming from the top 

[https://d36urhup7zbd7q.cloudfront.net/f77068df-cfa4-420e-8b83-7db24e8ff55d/Network_Dynamics_Logo_09_102620.format_png.resize_200x.png]<https://networkdynamics.com/>




Jonathon Musters
Network Dynamics Pty Ltd

Phone



1300 768 249 


Website



networkdynamics.com <http://networkdynamics.com>


Email



 jonat...@networkdynamics.com.au <mailto:jonat...@networkdynamics.com.au>









This email and any attachments are intended only for the use of the recipient 
and may be confidential and/or legally privileged. Network Dynamics disclaims 
liability for any errors, omissions, viruses, loss and/or damage arising from 
using, opening or transmitting this email. If you are not the intended 
recipient you must not use, interfere with, disclose, copy or retain this email 
and you should notify the sender immediately by return email or by contacting 
Network Dynamics by telephone on 1300 768 249. All views and opinions expressed 
in this email message are the personal opinions of the author and do not 
represent those of the company. No liability can be held for any damages, 
however caused, to any recipients of this message.









From: AusNOG 
mailto:ausnog-boun...@lists.ausnog.net>> on 
behalf of DaZZa mailto:dazzagi...@gmail.com>>
Date: Wednesday, November 8, 2023 at 1:10 PM
To: Jake Anderson mailto:ya...@vapourforge.com>>
Cc: ausnog@lists.ausnog.net<mailto:ausnog@lists.ausnog.net> 
mailto:ausnog@lists.ausnog.net>>
Subject: Re: [AusNOG] Optus downtime chat + affecting SMS verification to 
Telstra?
The ABC's rolling coverage reports "Some services across fixed and
mobile are now gradually being restored.", but it may take a "few
hours" for everything to recover.

Now we just get to wait for the report on the outage. it oughta be a doozy!

D

On Wed, 8 Nov 2023 at 12:15, Jake Anderson 
mailto:ya...@vapourforge.com>> wrote:
>
> Looks like they may getting some services back
> https://radar.cloudflare.com/as4804?dateStart=2023-11-07=2023-11-07
>
> On 8/11/2023 9:26 am, francisfi...@mailup.net<mailto:francisfi...@mailup.net> 
> wrote:
> > Morning all,
> > Hope the chaos isn't too hard on your work/family.
> > I have had trouble with a couple of SMS verifications coming through to me, 
> > my Telstra number. Is this related?
> >
> > Any general banter around the downtime would be fine too - looks like it 
> > all began at 4.07am AEDT?
> >
> > Cheers
> >
>
> ___
> AusNOG mailing list
> AusNOG@lists.ausnog.net<mailto:AusNOG@lists.ausnog.net>
> https://lists.ausnog.net/mailman/listinfo/ausnog



--
veg·e·tar·i·an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride
___
AusNOG mailing list
AusNOG@lists.ausnog.net<mailto:AusNOG@lists.ausnog.net>
https://lists.ausnog.net/mailman/listinfo/ausnog
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[Jonathon Musters]

Hey All,

This sounds much like what happened in Canada with Rogers –
https://en.wikipedia.org/wiki/2022_Rogers_Communications_outage

The same words seem to be coming from the top 



From: AusNOG  on behalf of DaZZa 

Date: Wednesday, November 8, 2023 at 1:10 PM
To: Jake Anderson 
Cc: ausnog@lists.ausnog.net 
Subject: Re: [AusNOG] Optus downtime chat + affecting SMS verification to 
Telstra?
The ABC's rolling coverage reports "Some services across fixed and
mobile are now gradually being restored.", but it may take a "few
hours" for everything to recover.

Now we just get to wait for the report on the outage. it oughta be a doozy!

D

On Wed, 8 Nov 2023 at 12:15, Jake Anderson  wrote:
>
> Looks like they may getting some services back
> https://radar.cloudflare.com/as4804?dateStart=2023-11-07=2023-11-07
>
> On 8/11/2023 9:26 am, francisfi...@mailup.net wrote:
> > Morning all,
> > Hope the chaos isn't too hard on your work/family.
> > I have had trouble with a couple of SMS verifications coming through to me, 
> > my Telstra number. Is this related?
> >
> > Any general banter around the downtime would be fine too - looks like it 
> > all began at 4.07am AEDT?
> >
> > Cheers
> >
>
> ___
> AusNOG mailing list
> AusNOG@lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog



--
veg·e·tar·i·an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[DaZZa]

The ABC's rolling coverage reports "Some services across fixed and
mobile are now gradually being restored.", but it may take a "few
hours" for everything to recover.

Now we just get to wait for the report on the outage. it oughta be a doozy!

D

On Wed, 8 Nov 2023 at 12:15, Jake Anderson  wrote:
>
> Looks like they may getting some services back
> https://radar.cloudflare.com/as4804?dateStart=2023-11-07=2023-11-07
>
> On 8/11/2023 9:26 am, francisfi...@mailup.net wrote:
> > Morning all,
> > Hope the chaos isn't too hard on your work/family.
> > I have had trouble with a couple of SMS verifications coming through to me, 
> > my Telstra number. Is this related?
> >
> > Any general banter around the downtime would be fine too - looks like it 
> > all began at 4.07am AEDT?
> >
> > Cheers
> >
>
> ___
> AusNOG mailing list
> AusNOG@lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog



-- 
veg·e·tar·i·an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[John Edwards]

Those who believe it's a BGP issue might want to review
https://www.cidr-report.org/cgi-bin/plota?file=%2fvar%2fdata%2fbgp%2fas2.0%2fbgp%2dactive%2etxt=Active%20BGP%20entries%20%28FIB%29=Active%20BGP%20entries%20%28FIB%29=step

This shows a global table getting dangerously close to a 1024K boundary
found in some router FIBs.

John


On Wed, 8 Nov 2023 at 11:57, Christopher Hawker 
wrote:

> They weren't appearing earlier so I'd say it's new. Smells like BGP has
> shat itself.
>
> - ​CH
> --
> *From:* AusNOG  on behalf of Tim Malone <
> tdmal...@gmail.com>
> *Sent:* Wednesday, November 8, 2023 12:16 PM
> *To:* Nick Jenkin 
> *Cc:* ausnog@lists.ausnog.net 
> *Subject:* Re: [AusNOG] Optus downtime chat + affecting SMS verification
> to Telstra?
>
> Cloudflare is also reporting BGP origin hijack warnings:
> https://radar.cloudflare.com/as7474?dateRange=2d
> (unsure if related though)
>
>
> On Wed, 8 Nov 2023 at 11:10, Nick Jenkin  wrote:
>
> Just a few announcements...
> https://radar.cloudflare.com/traffic/as4804?dateRange=2d
>
> On Wed, 8 Nov 2023 at 11:06, Ben Buxton  wrote:
>
>
>
> On Wed, 8 Nov 2023 at 10:14, DaZZa  wrote:
>
> Yeah, I'd be willing to bet that it's a change which wasn't thoroughly
> tested before being rolled out, and which had an inadequate backout
> plan.
>
>
> Also, "Our on-site technician is actively prioritising establishing a
> console connection.".
>
> I mean come on, it's nearly 2024 and a [major] telco does not have remote
> console access? Whilst I'm
> looking forward to enthusiastically reading the PM, I'll have to book a
> physio appointment in advance due to
> neck strain from all the head shaking it'll likely induce.
>
> BB
>
>
>
>
> Interestingly, my Optus mobile actually had a valid connection for a
> short time - wasn't able to actually DO anything, but was connected to
> the OPtus network - but it's now gone to "SOS" mode.
>
> D
>
> On Wed, 8 Nov 2023 at 10:01, John Edwards  wrote:
> >
> > The 4am Wednesday morning outage start looks suspiciously like a
> firmware upgrade window.
> >
> > I note that Optus devices where I am are showing "SoS" which indicates
> the tower is unable to reach the location register, which presumably is on
> a private network and indicative of a pretty major fault rather than just
> IP.
> >
> > John
> >
> >
> > On Wed, 8 Nov 2023 at 09:10, DaZZa  wrote:
> >>
> >> The Optus hamster finally died of old age.
> >>
> >> I would suggest your SMS issues would be caused by whoever is issuing
> >> the SMS using Optus - not so much by the Telstra end receiving it.
> >>
> >> Anecdotally, Optus enterprise/wholesale appears to be still functional
> >> - at least my link appears to be working fine - and my BGP
> >> advertisements are still being seen overseas - seems to be only NBN
> >> and mobile based services which are busted
> >>
> >> D
> >>
> >> On Wed, 8 Nov 2023 at 09:27,  wrote:
> >> >
> >> > Morning all,
> >> > Hope the chaos isn't too hard on your work/family.
> >> > I have had trouble with a couple of SMS verifications coming through
> to me, my Telstra number. Is this related?
> >> >
> >> > Any general banter around the downtime would be fine too - looks like
> it all began at 4.07am AEDT?
> >> >
> >> > Cheers
> >> >
> >> > --
> >> >
> >> >   francisfi...@mailup.net
> >> > ___
> >> > AusNOG mailing list
> >> > AusNOG@lists.ausnog.net
> >> > https://lists.ausnog.net/mailman/listinfo/ausnog
> >>
> >>
> >>
> >> --
> >> veg·e·tar·i·an:
> >> Ancient tribal slang for the village idiot who can't hunt, fish or ride
> >> ___
> >> AusNOG mailing list
> >> AusNOG@lists.ausnog.net
> >> https://lists.ausnog.net/mailman/listinfo/ausnog
>
>
>
> --
> veg·e·tar·i·an:
> Ancient tribal slang for the village idiot who can't hunt, fish or ride
> ___
> AusNOG mailing list
> AusNOG@lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
> ___
> AusNOG mailing list
> AusNOG@lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
> ___
> AusNOG mailing list
> AusNOG@lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
> ___
> AusNOG mailing list
> AusNOG@lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[Christopher Hawker]

They weren't appearing earlier so I'd say it's new. Smells like BGP has shat 
itself.

- ​CH

From: AusNOG  on behalf of Tim Malone 

Sent: Wednesday, November 8, 2023 12:16 PM
To: Nick Jenkin 
Cc: ausnog@lists.ausnog.net 
Subject: Re: [AusNOG] Optus downtime chat + affecting SMS verification to 
Telstra?

Cloudflare is also reporting BGP origin hijack warnings:
https://radar.cloudflare.com/as7474?dateRange=2d
(unsure if related though)


On Wed, 8 Nov 2023 at 11:10, Nick Jenkin 
mailto:njen...@gmail.com>> wrote:
Just a few announcements... 
https://radar.cloudflare.com/traffic/as4804?dateRange=2d

On Wed, 8 Nov 2023 at 11:06, Ben Buxton 
mailto:bb.aus...@bb.cactii.net>> wrote:


On Wed, 8 Nov 2023 at 10:14, DaZZa 
mailto:dazzagi...@gmail.com>> wrote:
Yeah, I'd be willing to bet that it's a change which wasn't thoroughly
tested before being rolled out, and which had an inadequate backout
plan.

Also, "Our on-site technician is actively prioritising establishing a console 
connection.".

I mean come on, it's nearly 2024 and a [major] telco does not have remote 
console access? Whilst I'm
looking forward to enthusiastically reading the PM, I'll have to book a physio 
appointment in advance due to
neck strain from all the head shaking it'll likely induce.

BB



Interestingly, my Optus mobile actually had a valid connection for a
short time - wasn't able to actually DO anything, but was connected to
the OPtus network - but it's now gone to "SOS" mode.

D

On Wed, 8 Nov 2023 at 10:01, John Edwards 
mailto:jaedwa...@gmail.com>> wrote:
>
> The 4am Wednesday morning outage start looks suspiciously like a firmware 
> upgrade window.
>
> I note that Optus devices where I am are showing "SoS" which indicates the 
> tower is unable to reach the location register, which presumably is on a 
> private network and indicative of a pretty major fault rather than just IP.
>
> John
>
>
> On Wed, 8 Nov 2023 at 09:10, DaZZa 
> mailto:dazzagi...@gmail.com>> wrote:
>>
>> The Optus hamster finally died of old age.
>>
>> I would suggest your SMS issues would be caused by whoever is issuing
>> the SMS using Optus - not so much by the Telstra end receiving it.
>>
>> Anecdotally, Optus enterprise/wholesale appears to be still functional
>> - at least my link appears to be working fine - and my BGP
>> advertisements are still being seen overseas - seems to be only NBN
>> and mobile based services which are busted
>>
>> D
>>
>> On Wed, 8 Nov 2023 at 09:27, 
>> mailto:francisfi...@mailup.net>> wrote:
>> >
>> > Morning all,
>> > Hope the chaos isn't too hard on your work/family.
>> > I have had trouble with a couple of SMS verifications coming through to 
>> > me, my Telstra number. Is this related?
>> >
>> > Any general banter around the downtime would be fine too - looks like it 
>> > all began at 4.07am AEDT?
>> >
>> > Cheers
>> >
>> > --
>> >
>> >   francisfi...@mailup.net<mailto:francisfi...@mailup.net>
>> > ___
>> > AusNOG mailing list
>> > AusNOG@lists.ausnog.net<mailto:AusNOG@lists.ausnog.net>
>> > https://lists.ausnog.net/mailman/listinfo/ausnog
>>
>>
>>
>> --
>> veg·e·tar·i·an:
>> Ancient tribal slang for the village idiot who can't hunt, fish or ride
>> ___
>> AusNOG mailing list
>> AusNOG@lists.ausnog.net<mailto:AusNOG@lists.ausnog.net>
>> https://lists.ausnog.net/mailman/listinfo/ausnog



--
veg·e·tar·i·an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride
___
AusNOG mailing list
AusNOG@lists.ausnog.net<mailto:AusNOG@lists.ausnog.net>
https://lists.ausnog.net/mailman/listinfo/ausnog
___
AusNOG mailing list
AusNOG@lists.ausnog.net<mailto:AusNOG@lists.ausnog.net>
https://lists.ausnog.net/mailman/listinfo/ausnog
___
AusNOG mailing list
AusNOG@lists.ausnog.net<mailto:AusNOG@lists.ausnog.net>
https://lists.ausnog.net/mailman/listinfo/ausnog
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[Tim Malone]

Cloudflare is also reporting BGP origin hijack warnings:
https://radar.cloudflare.com/as7474?dateRange=2d
(unsure if related though)


On Wed, 8 Nov 2023 at 11:10, Nick Jenkin  wrote:

> Just a few announcements...
> https://radar.cloudflare.com/traffic/as4804?dateRange=2d
>
> On Wed, 8 Nov 2023 at 11:06, Ben Buxton  wrote:
>
>>
>>
>> On Wed, 8 Nov 2023 at 10:14, DaZZa  wrote:
>>
>>> Yeah, I'd be willing to bet that it's a change which wasn't thoroughly
>>> tested before being rolled out, and which had an inadequate backout
>>> plan.
>>>
>>
>> Also, "Our on-site technician is actively prioritising establishing a
>> console connection.".
>>
>> I mean come on, it's nearly 2024 and a [major] telco does not have remote
>> console access? Whilst I'm
>> looking forward to enthusiastically reading the PM, I'll have to book a
>> physio appointment in advance due to
>> neck strain from all the head shaking it'll likely induce.
>>
>> BB
>>
>>
>>
>>>
>>> Interestingly, my Optus mobile actually had a valid connection for a
>>> short time - wasn't able to actually DO anything, but was connected to
>>> the OPtus network - but it's now gone to "SOS" mode.
>>>
>>> D
>>>
>>> On Wed, 8 Nov 2023 at 10:01, John Edwards  wrote:
>>> >
>>> > The 4am Wednesday morning outage start looks suspiciously like a
>>> firmware upgrade window.
>>> >
>>> > I note that Optus devices where I am are showing "SoS" which indicates
>>> the tower is unable to reach the location register, which presumably is on
>>> a private network and indicative of a pretty major fault rather than just
>>> IP.
>>> >
>>> > John
>>> >
>>> >
>>> > On Wed, 8 Nov 2023 at 09:10, DaZZa  wrote:
>>> >>
>>> >> The Optus hamster finally died of old age.
>>> >>
>>> >> I would suggest your SMS issues would be caused by whoever is issuing
>>> >> the SMS using Optus - not so much by the Telstra end receiving it.
>>> >>
>>> >> Anecdotally, Optus enterprise/wholesale appears to be still functional
>>> >> - at least my link appears to be working fine - and my BGP
>>> >> advertisements are still being seen overseas - seems to be only NBN
>>> >> and mobile based services which are busted
>>> >>
>>> >> D
>>> >>
>>> >> On Wed, 8 Nov 2023 at 09:27,  wrote:
>>> >> >
>>> >> > Morning all,
>>> >> > Hope the chaos isn't too hard on your work/family.
>>> >> > I have had trouble with a couple of SMS verifications coming
>>> through to me, my Telstra number. Is this related?
>>> >> >
>>> >> > Any general banter around the downtime would be fine too - looks
>>> like it all began at 4.07am AEDT?
>>> >> >
>>> >> > Cheers
>>> >> >
>>> >> > --
>>> >> >
>>> >> >   francisfi...@mailup.net
>>> >> > ___
>>> >> > AusNOG mailing list
>>> >> > AusNOG@lists.ausnog.net
>>> >> > https://lists.ausnog.net/mailman/listinfo/ausnog
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> veg·e·tar·i·an:
>>> >> Ancient tribal slang for the village idiot who can't hunt, fish or
>>> ride
>>> >> ___
>>> >> AusNOG mailing list
>>> >> AusNOG@lists.ausnog.net
>>> >> https://lists.ausnog.net/mailman/listinfo/ausnog
>>>
>>>
>>>
>>> --
>>> veg·e·tar·i·an:
>>> Ancient tribal slang for the village idiot who can't hunt, fish or ride
>>> ___
>>> AusNOG mailing list
>>> AusNOG@lists.ausnog.net
>>> https://lists.ausnog.net/mailman/listinfo/ausnog
>>>
>> ___
>> AusNOG mailing list
>> AusNOG@lists.ausnog.net
>> https://lists.ausnog.net/mailman/listinfo/ausnog
>>
> ___
> AusNOG mailing list
> AusNOG@lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[Jake Anderson]

Looks like they may getting some services back
https://radar.cloudflare.com/as4804?dateStart=2023-11-07=2023-11-07

On 8/11/2023 9:26 am, francisfi...@mailup.net wrote:

Morning all,
Hope the chaos isn't too hard on your work/family.
I have had trouble with a couple of SMS verifications coming through to me, my 
Telstra number. Is this related?

Any general banter around the downtime would be fine too - looks like it all 
began at 4.07am AEDT?

Cheers



___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[Mark Andrews]

I’m wondering if it is leaking over to Vodafone as International Roaming has 
failed for me here in Prague at about the same time.  All I’m getting is “no 
servíce” even after forcing the carrier to Vodafone CZ which I’ve been using 
the last 8 days. 

-- 
Mark Andrews

> On 8 Nov 2023, at 01:10, Nick Jenkin  wrote:
> 
> 
> Just a few announcements... 
> https://radar.cloudflare.com/traffic/as4804?dateRange=2d
> 
>> On Wed, 8 Nov 2023 at 11:06, Ben Buxton  wrote:
>> 
>> 
>>> On Wed, 8 Nov 2023 at 10:14, DaZZa  wrote:
>>> Yeah, I'd be willing to bet that it's a change which wasn't thoroughly
>>> tested before being rolled out, and which had an inadequate backout
>>> plan.
>> 
>> Also, "Our on-site technician is actively prioritising establishing a 
>> console connection.".
>> 
>> I mean come on, it's nearly 2024 and a [major] telco does not have remote 
>> console access? Whilst I'm
>> looking forward to enthusiastically reading the PM, I'll have to book a 
>> physio appointment in advance due to 
>> neck strain from all the head shaking it'll likely induce.
>> 
>> BB
>> 
>>  
>>> 
>>> Interestingly, my Optus mobile actually had a valid connection for a
>>> short time - wasn't able to actually DO anything, but was connected to
>>> the OPtus network - but it's now gone to "SOS" mode.
>>> 
>>> D
>>> 
>>> On Wed, 8 Nov 2023 at 10:01, John Edwards  wrote:
>>> >
>>> > The 4am Wednesday morning outage start looks suspiciously like a firmware 
>>> > upgrade window.
>>> >
>>> > I note that Optus devices where I am are showing "SoS" which indicates 
>>> > the tower is unable to reach the location register, which presumably is 
>>> > on a private network and indicative of a pretty major fault rather than 
>>> > just IP.
>>> >
>>> > John
>>> >
>>> >
>>> > On Wed, 8 Nov 2023 at 09:10, DaZZa  wrote:
>>> >>
>>> >> The Optus hamster finally died of old age.
>>> >>
>>> >> I would suggest your SMS issues would be caused by whoever is issuing
>>> >> the SMS using Optus - not so much by the Telstra end receiving it.
>>> >>
>>> >> Anecdotally, Optus enterprise/wholesale appears to be still functional
>>> >> - at least my link appears to be working fine - and my BGP
>>> >> advertisements are still being seen overseas - seems to be only NBN
>>> >> and mobile based services which are busted
>>> >>
>>> >> D
>>> >>
>>> >> On Wed, 8 Nov 2023 at 09:27,  wrote:
>>> >> >
>>> >> > Morning all,
>>> >> > Hope the chaos isn't too hard on your work/family.
>>> >> > I have had trouble with a couple of SMS verifications coming through 
>>> >> > to me, my Telstra number. Is this related?
>>> >> >
>>> >> > Any general banter around the downtime would be fine too - looks like 
>>> >> > it all began at 4.07am AEDT?
>>> >> >
>>> >> > Cheers
>>> >> >
>>> >> > --
>>> >> >
>>> >> >   francisfi...@mailup.net
>>> >> > ___
>>> >> > AusNOG mailing list
>>> >> > AusNOG@lists.ausnog.net
>>> >> > https://lists.ausnog.net/mailman/listinfo/ausnog
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> veg·e·tar·i·an:
>>> >> Ancient tribal slang for the village idiot who can't hunt, fish or ride
>>> >> ___
>>> >> AusNOG mailing list
>>> >> AusNOG@lists.ausnog.net
>>> >> https://lists.ausnog.net/mailman/listinfo/ausnog
>>> 
>>> 
>>> 
>>> -- 
>>> veg·e·tar·i·an:
>>> Ancient tribal slang for the village idiot who can't hunt, fish or ride
>>> ___
>>> AusNOG mailing list
>>> AusNOG@lists.ausnog.net
>>> https://lists.ausnog.net/mailman/listinfo/ausnog
>> ___
>> AusNOG mailing list
>> AusNOG@lists.ausnog.net
>> https://lists.ausnog.net/mailman/listinfo/ausnog
> ___
> AusNOG mailing list
> AusNOG@lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[Nick Jenkin]

Just a few announcements...
https://radar.cloudflare.com/traffic/as4804?dateRange=2d

On Wed, 8 Nov 2023 at 11:06, Ben Buxton  wrote:

>
>
> On Wed, 8 Nov 2023 at 10:14, DaZZa  wrote:
>
>> Yeah, I'd be willing to bet that it's a change which wasn't thoroughly
>> tested before being rolled out, and which had an inadequate backout
>> plan.
>>
>
> Also, "Our on-site technician is actively prioritising establishing a
> console connection.".
>
> I mean come on, it's nearly 2024 and a [major] telco does not have remote
> console access? Whilst I'm
> looking forward to enthusiastically reading the PM, I'll have to book a
> physio appointment in advance due to
> neck strain from all the head shaking it'll likely induce.
>
> BB
>
>
>
>>
>> Interestingly, my Optus mobile actually had a valid connection for a
>> short time - wasn't able to actually DO anything, but was connected to
>> the OPtus network - but it's now gone to "SOS" mode.
>>
>> D
>>
>> On Wed, 8 Nov 2023 at 10:01, John Edwards  wrote:
>> >
>> > The 4am Wednesday morning outage start looks suspiciously like a
>> firmware upgrade window.
>> >
>> > I note that Optus devices where I am are showing "SoS" which indicates
>> the tower is unable to reach the location register, which presumably is on
>> a private network and indicative of a pretty major fault rather than just
>> IP.
>> >
>> > John
>> >
>> >
>> > On Wed, 8 Nov 2023 at 09:10, DaZZa  wrote:
>> >>
>> >> The Optus hamster finally died of old age.
>> >>
>> >> I would suggest your SMS issues would be caused by whoever is issuing
>> >> the SMS using Optus - not so much by the Telstra end receiving it.
>> >>
>> >> Anecdotally, Optus enterprise/wholesale appears to be still functional
>> >> - at least my link appears to be working fine - and my BGP
>> >> advertisements are still being seen overseas - seems to be only NBN
>> >> and mobile based services which are busted
>> >>
>> >> D
>> >>
>> >> On Wed, 8 Nov 2023 at 09:27,  wrote:
>> >> >
>> >> > Morning all,
>> >> > Hope the chaos isn't too hard on your work/family.
>> >> > I have had trouble with a couple of SMS verifications coming through
>> to me, my Telstra number. Is this related?
>> >> >
>> >> > Any general banter around the downtime would be fine too - looks
>> like it all began at 4.07am AEDT?
>> >> >
>> >> > Cheers
>> >> >
>> >> > --
>> >> >
>> >> >   francisfi...@mailup.net
>> >> > ___
>> >> > AusNOG mailing list
>> >> > AusNOG@lists.ausnog.net
>> >> > https://lists.ausnog.net/mailman/listinfo/ausnog
>> >>
>> >>
>> >>
>> >> --
>> >> veg·e·tar·i·an:
>> >> Ancient tribal slang for the village idiot who can't hunt, fish or ride
>> >> ___
>> >> AusNOG mailing list
>> >> AusNOG@lists.ausnog.net
>> >> https://lists.ausnog.net/mailman/listinfo/ausnog
>>
>>
>>
>> --
>> veg·e·tar·i·an:
>> Ancient tribal slang for the village idiot who can't hunt, fish or ride
>> ___
>> AusNOG mailing list
>> AusNOG@lists.ausnog.net
>> https://lists.ausnog.net/mailman/listinfo/ausnog
>>
> ___
> AusNOG mailing list
> AusNOG@lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[Ben Buxton]

On Wed, 8 Nov 2023 at 10:14, DaZZa  wrote:

> Yeah, I'd be willing to bet that it's a change which wasn't thoroughly
> tested before being rolled out, and which had an inadequate backout
> plan.
>

Also, "Our on-site technician is actively prioritising establishing a
console connection.".

I mean come on, it's nearly 2024 and a [major] telco does not have remote
console access? Whilst I'm
looking forward to enthusiastically reading the PM, I'll have to book a
physio appointment in advance due to
neck strain from all the head shaking it'll likely induce.

BB



>
> Interestingly, my Optus mobile actually had a valid connection for a
> short time - wasn't able to actually DO anything, but was connected to
> the OPtus network - but it's now gone to "SOS" mode.
>
> D
>
> On Wed, 8 Nov 2023 at 10:01, John Edwards  wrote:
> >
> > The 4am Wednesday morning outage start looks suspiciously like a
> firmware upgrade window.
> >
> > I note that Optus devices where I am are showing "SoS" which indicates
> the tower is unable to reach the location register, which presumably is on
> a private network and indicative of a pretty major fault rather than just
> IP.
> >
> > John
> >
> >
> > On Wed, 8 Nov 2023 at 09:10, DaZZa  wrote:
> >>
> >> The Optus hamster finally died of old age.
> >>
> >> I would suggest your SMS issues would be caused by whoever is issuing
> >> the SMS using Optus - not so much by the Telstra end receiving it.
> >>
> >> Anecdotally, Optus enterprise/wholesale appears to be still functional
> >> - at least my link appears to be working fine - and my BGP
> >> advertisements are still being seen overseas - seems to be only NBN
> >> and mobile based services which are busted
> >>
> >> D
> >>
> >> On Wed, 8 Nov 2023 at 09:27,  wrote:
> >> >
> >> > Morning all,
> >> > Hope the chaos isn't too hard on your work/family.
> >> > I have had trouble with a couple of SMS verifications coming through
> to me, my Telstra number. Is this related?
> >> >
> >> > Any general banter around the downtime would be fine too - looks like
> it all began at 4.07am AEDT?
> >> >
> >> > Cheers
> >> >
> >> > --
> >> >
> >> >   francisfi...@mailup.net
> >> > ___
> >> > AusNOG mailing list
> >> > AusNOG@lists.ausnog.net
> >> > https://lists.ausnog.net/mailman/listinfo/ausnog
> >>
> >>
> >>
> >> --
> >> veg·e·tar·i·an:
> >> Ancient tribal slang for the village idiot who can't hunt, fish or ride
> >> ___
> >> AusNOG mailing list
> >> AusNOG@lists.ausnog.net
> >> https://lists.ausnog.net/mailman/listinfo/ausnog
>
>
>
> --
> veg·e·tar·i·an:
> Ancient tribal slang for the village idiot who can't hunt, fish or ride
> ___
> AusNOG mailing list
> AusNOG@lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[DaZZa]

I loved the scramble to avoid saying "We f* up". And the dodge
when asked "How many staff are on-shore and how manby overseas". I
laughed

She's a talking head. I didn't expect any information from her - what
I took out of it is that 7 hours after this started, they *still*
don't know what caused it, or how to fix it.

D

On Wed, 8 Nov 2023 at 10:55, Robert Hudson  wrote:
>
> Gotta say, that interview didn't give me any confidence at all.  It makes me 
> wonder if the CEO even knows what Optus does as its core business, or has had 
> any sort of media training on how to communicate with customers and the 
> public.
>
> "We'd like the media to help by directing people to our messaging" - that 
> they can't access...
>
> On Wed, 8 Nov 2023 at 10:33, Bradley Amm  wrote:
>>
>> https://www.abc.net.au/news/2023-11-08/optus-outage-live-blog/103076996
>>
>> Get Outlook for iOS
>> 
>> From: AusNOG  on behalf of DaZZa 
>> 
>> Sent: Wednesday, November 8, 2023 6:39:22 AM
>> To: francisfi...@mailup.net 
>> Cc: AusNOG@lists.ausnog.net 
>> Subject: Re: [AusNOG] Optus downtime chat + affecting SMS verification to 
>> Telstra?
>>
>> The Optus hamster finally died of old age.
>>
>> I would suggest your SMS issues would be caused by whoever is issuing
>> the SMS using Optus - not so much by the Telstra end receiving it.
>>
>> Anecdotally, Optus enterprise/wholesale appears to be still functional
>> - at least my link appears to be working fine - and my BGP
>> advertisements are still being seen overseas - seems to be only NBN
>> and mobile based services which are busted
>>
>> D
>>
>> On Wed, 8 Nov 2023 at 09:27,  wrote:
>> >
>> > Morning all,
>> > Hope the chaos isn't too hard on your work/family.
>> > I have had trouble with a couple of SMS verifications coming through to 
>> > me, my Telstra number. Is this related?
>> >
>> > Any general banter around the downtime would be fine too - looks like it 
>> > all began at 4.07am AEDT?
>> >
>> > Cheers
>> >
>> > --
>> >
>> >   francisfi...@mailup.net
>> > ___
>> > AusNOG mailing list
>> > AusNOG@lists.ausnog.net
>> > https://lists.ausnog.net/mailman/listinfo/ausnog
>>
>>
>>
>> --
>> veg·e·tar·i·an:
>> Ancient tribal slang for the village idiot who can't hunt, fish or ride
>> ___
>> AusNOG mailing list
>> AusNOG@lists.ausnog.net
>> https://lists.ausnog.net/mailman/listinfo/ausnog
>> ___
>> AusNOG mailing list
>> AusNOG@lists.ausnog.net
>> https://lists.ausnog.net/mailman/listinfo/ausnog



-- 
veg·e·tar·i·an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[Robert Hudson]

Gotta say, that interview didn't give me any confidence at all.  It makes
me wonder if the CEO even knows what Optus does as its core business, or
has had any sort of media training on how to communicate with customers and
the public.

"We'd like the media to help by directing people to our messaging" - that
they can't access...

On Wed, 8 Nov 2023 at 10:33, Bradley Amm  wrote:

> https://www.abc.net.au/news/2023-11-08/optus-outage-live-blog/103076996
>
> Get Outlook for iOS <https://aka.ms/o0ukef>
> --
> *From:* AusNOG  on behalf of DaZZa <
> dazzagi...@gmail.com>
> *Sent:* Wednesday, November 8, 2023 6:39:22 AM
> *To:* francisfi...@mailup.net 
> *Cc:* AusNOG@lists.ausnog.net 
> *Subject:* Re: [AusNOG] Optus downtime chat + affecting SMS verification
> to Telstra?
>
> The Optus hamster finally died of old age.
>
> I would suggest your SMS issues would be caused by whoever is issuing
> the SMS using Optus - not so much by the Telstra end receiving it.
>
> Anecdotally, Optus enterprise/wholesale appears to be still functional
> - at least my link appears to be working fine - and my BGP
> advertisements are still being seen overseas - seems to be only NBN
> and mobile based services which are busted
>
> D
>
> On Wed, 8 Nov 2023 at 09:27,  wrote:
> >
> > Morning all,
> > Hope the chaos isn't too hard on your work/family.
> > I have had trouble with a couple of SMS verifications coming through to
> me, my Telstra number. Is this related?
> >
> > Any general banter around the downtime would be fine too - looks like it
> all began at 4.07am AEDT?
> >
> > Cheers
> >
> > --
> >
> >   francisfi...@mailup.net
> > ___
> > AusNOG mailing list
> > AusNOG@lists.ausnog.net
> > https://lists.ausnog.net/mailman/listinfo/ausnog
>
>
>
> --
> veg·e·tar·i·an:
> Ancient tribal slang for the village idiot who can't hunt, fish or ride
> ___
> AusNOG mailing list
> AusNOG@lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
> ___
> AusNOG mailing list
> AusNOG@lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[yahoo]

I happened to be looking at my phone when it happened. It said words to the 
effect of 'the carrier has turned off mobile data for a short time" with a 
little triangle symbol that I've never seen before.I guess it wasn't wrong 
depending on how you define shortSent from my Galaxy
 Original message From: francisfi...@mailup.net Date: 8/11/23  
9:27 am  (GMT+10:00) To: AusNOG@lists.ausnog.net Subject: [AusNOG] Optus 
downtime chat + affecting SMS verification to
Telstra? Morning all,Hope the chaos isn't too hard on your work/family. 
I have had trouble with a couple of SMS verifications coming through to me, my 
Telstra number. Is this related?Any general banter around the downtime would be 
fine too - looks like it all began at 4.07am AEDT?Cheers--     
francisfides@mailup.net___AusNOG 
mailing 
listAusNOG@lists.ausnog.nethttps://lists.ausnog.net/mailman/listinfo/ausnog___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[Bradley Amm]

Maybe they were annoyed at glen maxwell getting 200

Get Outlook for iOS

From: AusNOG  on behalf of 
francisfi...@mailup.net 
Sent: Wednesday, November 8, 2023 6:26:26 AM
To: AusNOG@lists.ausnog.net 
Subject: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

Morning all,
Hope the chaos isn't too hard on your work/family.
I have had trouble with a couple of SMS verifications coming through to me, my 
Telstra number. Is this related?

Any general banter around the downtime would be fine too - looks like it all 
began at 4.07am AEDT?

Cheers

--

  francisfi...@mailup.net
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[Bradley Amm]

https://www.abc.net.au/news/2023-11-08/optus-outage-live-blog/103076996

Get Outlook for iOS<https://aka.ms/o0ukef>

From: AusNOG  on behalf of DaZZa 

Sent: Wednesday, November 8, 2023 6:39:22 AM
To: francisfi...@mailup.net 
Cc: AusNOG@lists.ausnog.net 
Subject: Re: [AusNOG] Optus downtime chat + affecting SMS verification to 
Telstra?

The Optus hamster finally died of old age.

I would suggest your SMS issues would be caused by whoever is issuing
the SMS using Optus - not so much by the Telstra end receiving it.

Anecdotally, Optus enterprise/wholesale appears to be still functional
- at least my link appears to be working fine - and my BGP
advertisements are still being seen overseas - seems to be only NBN
and mobile based services which are busted

D

On Wed, 8 Nov 2023 at 09:27,  wrote:
>
> Morning all,
> Hope the chaos isn't too hard on your work/family.
> I have had trouble with a couple of SMS verifications coming through to me, 
> my Telstra number. Is this related?
>
> Any general banter around the downtime would be fine too - looks like it all 
> began at 4.07am AEDT?
>
> Cheers
>
> --
>
>   francisfi...@mailup.net
> ___
> AusNOG mailing list
> AusNOG@lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog



--
veg·e·tar·i·an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[Ben Buxton]

On Wed, 8 Nov 2023 at 09:29, Joseph Goldman  wrote:

>
> I would also have to imagine the outage is a bad config pushed out -
> surely a hardware failure or fibre cut wouldn't cause such a big outage.
> Bad day to have your name on the last push :/.
>

In a blameless culture, I'l probably consider it a badge of honour to have
my name on the push, but this is Optus so some engineer will be thrown
under the bus.

(In an even better culture though, no human would be manually performing
any push...but again this is Optus...)

BB



>
> -- Original Message --
> From: francisfi...@mailup.net
> To: AusNOG@lists.ausnog.net
> Sent: 8/11/2023 9:26:26 AM
> Subject: [AusNOG] Optus downtime chat + affecting SMS verification to
> Telstra?
>
> >Morning all,
> >Hope the chaos isn't too hard on your work/family.
> >I have had trouble with a couple of SMS verifications coming through to
> me, my Telstra number. Is this related?
> >
> >Any general banter around the downtime would be fine too - looks like it
> all began at 4.07am AEDT?
> >
> >Cheers
> >
> >--
> >
> >francisfi...@mailup.net
> >___
> >AusNOG mailing list
> >AusNOG@lists.ausnog.net
> >https://lists.ausnog.net/mailman/listinfo/ausnog
> >
>
> ___
> AusNOG mailing list
> AusNOG@lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[DaZZa]

Yeah, I'd be willing to bet that it's a change which wasn't thoroughly
tested before being rolled out, and which had an inadequate backout
plan.

Interestingly, my Optus mobile actually had a valid connection for a
short time - wasn't able to actually DO anything, but was connected to
the OPtus network - but it's now gone to "SOS" mode.

D

On Wed, 8 Nov 2023 at 10:01, John Edwards  wrote:
>
> The 4am Wednesday morning outage start looks suspiciously like a firmware 
> upgrade window.
>
> I note that Optus devices where I am are showing "SoS" which indicates the 
> tower is unable to reach the location register, which presumably is on a 
> private network and indicative of a pretty major fault rather than just IP.
>
> John
>
>
> On Wed, 8 Nov 2023 at 09:10, DaZZa  wrote:
>>
>> The Optus hamster finally died of old age.
>>
>> I would suggest your SMS issues would be caused by whoever is issuing
>> the SMS using Optus - not so much by the Telstra end receiving it.
>>
>> Anecdotally, Optus enterprise/wholesale appears to be still functional
>> - at least my link appears to be working fine - and my BGP
>> advertisements are still being seen overseas - seems to be only NBN
>> and mobile based services which are busted
>>
>> D
>>
>> On Wed, 8 Nov 2023 at 09:27,  wrote:
>> >
>> > Morning all,
>> > Hope the chaos isn't too hard on your work/family.
>> > I have had trouble with a couple of SMS verifications coming through to 
>> > me, my Telstra number. Is this related?
>> >
>> > Any general banter around the downtime would be fine too - looks like it 
>> > all began at 4.07am AEDT?
>> >
>> > Cheers
>> >
>> > --
>> >
>> >   francisfi...@mailup.net
>> > ___
>> > AusNOG mailing list
>> > AusNOG@lists.ausnog.net
>> > https://lists.ausnog.net/mailman/listinfo/ausnog
>>
>>
>>
>> --
>> veg·e·tar·i·an:
>> Ancient tribal slang for the village idiot who can't hunt, fish or ride
>> ___
>> AusNOG mailing list
>> AusNOG@lists.ausnog.net
>> https://lists.ausnog.net/mailman/listinfo/ausnog



-- 
veg·e·tar·i·an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[John Edwards]

The 4am Wednesday morning outage start looks suspiciously like a firmware
upgrade window.

I note that Optus devices where I am are showing "SoS" which indicates the
tower is unable to reach the location register, which presumably is on a
private network and indicative of a pretty major fault rather than just IP.

John


On Wed, 8 Nov 2023 at 09:10, DaZZa  wrote:

> The Optus hamster finally died of old age.
>
> I would suggest your SMS issues would be caused by whoever is issuing
> the SMS using Optus - not so much by the Telstra end receiving it.
>
> Anecdotally, Optus enterprise/wholesale appears to be still functional
> - at least my link appears to be working fine - and my BGP
> advertisements are still being seen overseas - seems to be only NBN
> and mobile based services which are busted
>
> D
>
> On Wed, 8 Nov 2023 at 09:27,  wrote:
> >
> > Morning all,
> > Hope the chaos isn't too hard on your work/family.
> > I have had trouble with a couple of SMS verifications coming through to
> me, my Telstra number. Is this related?
> >
> > Any general banter around the downtime would be fine too - looks like it
> all began at 4.07am AEDT?
> >
> > Cheers
> >
> > --
> >
> >   francisfi...@mailup.net
> > ___
> > AusNOG mailing list
> > AusNOG@lists.ausnog.net
> > https://lists.ausnog.net/mailman/listinfo/ausnog
>
>
>
> --
> veg·e·tar·i·an:
> Ancient tribal slang for the village idiot who can't hunt, fish or ride
> ___
> AusNOG mailing list
> AusNOG@lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[DaZZa]

The Optus hamster finally died of old age.

I would suggest your SMS issues would be caused by whoever is issuing
the SMS using Optus - not so much by the Telstra end receiving it.

Anecdotally, Optus enterprise/wholesale appears to be still functional
- at least my link appears to be working fine - and my BGP
advertisements are still being seen overseas - seems to be only NBN
and mobile based services which are busted

D

On Wed, 8 Nov 2023 at 09:27,  wrote:
>
> Morning all,
> Hope the chaos isn't too hard on your work/family.
> I have had trouble with a couple of SMS verifications coming through to me, 
> my Telstra number. Is this related?
>
> Any general banter around the downtime would be fine too - looks like it all 
> began at 4.07am AEDT?
>
> Cheers
>
> --
>
>   francisfi...@mailup.net
> ___
> AusNOG mailing list
> AusNOG@lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog



-- 
veg·e·tar·i·an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[Shaun Coffey]

This was the update a bit after 7:30:

The suspected root cause of the issue lies with the route reflectors, which
are currently handling an excessive number of routes, leading to session
shutdown and a complete traffic halt. Our on-site technician is actively
prioritising establishing a console connection. Rest assured that said
technician is also being provided additional technical support remotely.
Kindly expect next update to be provided by 9AM AEDT if not sooner.


On Wed, 8 Nov 2023 at 9:29 am, Joseph Goldman  wrote:

> Was actually wondering why there wasn't a thread.
>
> My guess would be for 2FA SMS's not coming through, that the SMS
> provider the software/site is using relies on Optus to send the message.
>
> I would also have to imagine the outage is a bad config pushed out -
> surely a hardware failure or fibre cut wouldn't cause such a big outage.
> Bad day to have your name on the last push :/.
>
> -- Original Message --
> From: francisfi...@mailup.net
> To: AusNOG@lists.ausnog.net
> Sent: 8/11/2023 9:26:26 AM
> Subject: [AusNOG] Optus downtime chat + affecting SMS verification to
> Telstra?
>
> >Morning all,
> >Hope the chaos isn't too hard on your work/family.
> >I have had trouble with a couple of SMS verifications coming through to
> me, my Telstra number. Is this related?
> >
> >Any general banter around the downtime would be fine too - looks like it
> all began at 4.07am AEDT?
> >
> >Cheers
> >
> >--
> >
> >francisfi...@mailup.net
> >___
> >AusNOG mailing list
> >AusNOG@lists.ausnog.net
> >https://lists.ausnog.net/mailman/listinfo/ausnog
> >
>
> ___
> AusNOG mailing list
> AusNOG@lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[Joseph Goldman]

Was actually wondering why there wasn't a thread.

My guess would be for 2FA SMS's not coming through, that the SMS 
provider the software/site is using relies on Optus to send the message.


I would also have to imagine the outage is a bad config pushed out - 
surely a hardware failure or fibre cut wouldn't cause such a big outage. 
Bad day to have your name on the last push :/.


-- Original Message --
From: francisfi...@mailup.net
To: AusNOG@lists.ausnog.net
Sent: 8/11/2023 9:26:26 AM
Subject: [AusNOG] Optus downtime chat + affecting SMS verification to 
Telstra?



Morning all,
Hope the chaos isn't too hard on your work/family.
I have had trouble with a couple of SMS verifications coming through to me, my 
Telstra number. Is this related?

Any general banter around the downtime would be fine too - looks like it all 
began at 4.07am AEDT?

Cheers

--

francisfi...@mailup.net
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog



___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog