[AusNOG] Vocus Datacentre outage.

[Chad Kelly]

Thought I would create a new thread on this, as the issue wasn't related to 
AussieBB.
It is looking like this Melbourne outage is similar to the Sydney outage back 
in 2018, in that it appears to have been yet again caused by an issue with 
their UPS systems yet again.
I'll leave this hear 
https://www.arnnet.com.au/article/633314/servers-australia-uproots-services-after-vocus-data-centre-outage/
 and this article on today's outage as well.
https://www.itnews.com.au/news/vocus-melbourne-data-centre-hit-by-power-outage-591987

Regards Chad.



Chad Kelly
Manager
CPK Web Services
Phone 03 52730246
Web https://www.cpkws.com.au
[cid:image001.jpg@01D955D7.3EA59970]

___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Vocus Datacentre outage.

[DaZZa]

Doesn't say a whole lot about Vocus' data center design methodology, nor
their periodic contingency testing.

They're apparently replacing the entire UPS on Friday - guess we can all
expect more outages then when they stuff that up.

D

On Mon, 13 Mar 2023 at 18:30, Chad Kelly  wrote:

> Thought I would create a new thread on this, as the issue wasn’t related
> to AussieBB.
>
> It is looking like this Melbourne outage is similar to the Sydney outage
> back in 2018, in that it appears to have been yet again caused by an issue
> with their UPS systems yet again.
>
> I’ll leave this hear
> https://www.arnnet.com.au/article/633314/servers-australia-uproots-services-after-vocus-data-centre-outage/
> and this article on today’s outage as well.
>
>
> https://www.itnews.com.au/news/vocus-melbourne-data-centre-hit-by-power-outage-591987
>
>
>
> Regards Chad.
>
>
>
>
>
>
>
> Chad Kelly
>
> Manager
>
> CPK Web Services
>
> Phone 03 52730246
>
> Web https://www.cpkws.com.au
>
>
> ___
> AusNOG mailing list
> AusNOG@lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>


-- 
veg·e·tar·i·an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Vocus Datacentre outage.

[Bradley Amm]

Friends like let friends put gear in a Vocus dtacenter. Only NextDC or Equinix :) Sent from my iPhoneOn 14 Mar 2023, at 7:08 am, DaZZa  wrote:Doesn't say a whole lot about Vocus' data center design methodology, nor their periodic contingency testing.They're apparently replacing the entire UPS on Friday - guess we can all expect more outages then when they stuff that up.DOn Mon, 13 Mar 2023 at 18:30, Chad Kelly  wrote:







Thought I would create a new thread on this, as the issue wasn’t related to AussieBB.

It is looking like this Melbourne outage is similar to the Sydney outage back in 2018, in that it appears to have been yet again caused by an issue with their UPS systems yet again.
I’ll leave this hear 
https://www.arnnet.com.au/article/633314/servers-australia-uproots-services-after-vocus-data-centre-outage/ and this article on today’s outage as well.

https://www.itnews.com.au/news/vocus-melbourne-data-centre-hit-by-power-outage-591987
 
Regards Chad. 
 
 
 
Chad Kelly

Manager

CPK Web Services

Phone 03 52730246
Web https://www.cpkws.com.au

 



___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog
-- veg·e·tar·i·an:Ancient tribal slang for the village idiot who can't hunt, fish or ride
___AusNOG mailing listAusNOG@lists.ausnog.nethttps://lists.ausnog.net/mailman/listinfo/ausnog___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Vocus Datacentre outage.

[Matt Perkins]

Yea because Eqinix has never had a power outage. :)

I think it's a little premature to judge at the moment. Doesn't sound 
good based on the current info but we have all had bad days im sure.


Matt


On 14/3/2023 11:06 am, Bradley Amm wrote:
Friends like let friends put gear in a Vocus dtacenter. Only NextDC or 
Equinix :)


Sent from my iPhone


On 14 Mar 2023, at 7:08 am, DaZZa  wrote:


Doesn't say a whole lot about Vocus' data center design methodology, 
nor their periodic contingency testing.


They're apparently replacing the entire UPS on Friday - guess we can 
all expect more outages then when they stuff that up.


D

On Mon, 13 Mar 2023 at 18:30, Chad Kelly  wrote:

Thought I would create a new thread on this, as the issue wasn’t
related to AussieBB.

It is looking like this Melbourne outage is similar to the Sydney
outage back in 2018, in that it appears to have been yet again
caused by an issue with their UPS systems yet again.

I’ll leave this hear

https://www.arnnet.com.au/article/633314/servers-australia-uproots-services-after-vocus-data-centre-outage/
and this article on today’s outage as well.


https://www.itnews.com.au/news/vocus-melbourne-data-centre-hit-by-power-outage-591987

Regards Chad.

Chad Kelly

Manager

CPK Web Services

Phone 03 52730246

Web https://www.cpkws.com.au



___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog



--
veg·e·tar·i·an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog


___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog


--
** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL

/* Matt Perkins
Direct 02 8916 8101 Spectrum Networks Ptd. Ltd.
Office 1300 133 299m...@spectrum.com.au
ABN 66 090 112 913  Level 6, 350 George Street Sydney 2000

*/
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Vocus Datacentre outage.

[Nathan Brookfield]

I’ve never had an A+B power rail failure at Equinix in 18 years I’ve been in 
there facilities, TOUCH WOOD!!!

Nathan Brookfield | VK2NAB
General Manager

p: 1300 592 330 | m: 0412 266 008
e: nathan.brookfi...@iperium.com.au | 
w: iperium.com.au

Suite 4.02, 189 Kent Street Sydney NSW 2000
[cid:image001.png@01D95667.DCB722B0]
Your Telco Team
DISCLAIMER: This document is intended solely for the named addressee. This 
electronic communication, which includes any files or attachments thereto, 
contains proprietary or confidential information and may be privileged and 
otherwise protected under copyright or other applicable intellectual property 
laws. The use, disclosure, copying or distribution of any of the information 
contained in this document, by any person other than the addressee, is strictly 
prohibited. If you received this document in error, please contact the sender 
immediately and delete all the material from any computer. Confidentiality and 
legal privilege are not waived or lost by reason of mistaken delivery to you. 
Any views or opinions presented are solely those of the author and do not 
necessarily represent those of Iperium.

WARNING: Computer viruses can be transmitted via email. The recipient should 
check this email and any attachments for the presence of viruses. Iperium 
accepts no liability for any damage caused by any virus transmitted by this 
email.
From: AusNOG  On Behalf Of Matt Perkins
Sent: Tuesday, 14 March 2023 11:18 AM
To: ausnog@lists.ausnog.net
Subject: Re: [AusNOG] Vocus Datacentre outage.


Yea because Eqinix has never had a power outage. :)

I think it's a little premature to judge at the moment. Doesn't sound good 
based on the current info but we have all had bad days im sure.

Matt


On 14/3/2023 11:06 am, Bradley Amm wrote:
Friends like let friends put gear in a Vocus dtacenter. Only NextDC or Equinix 
:)
Sent from my iPhone


On 14 Mar 2023, at 7:08 am, DaZZa 
 wrote:

Doesn't say a whole lot about Vocus' data center design methodology, nor their 
periodic contingency testing.

They're apparently replacing the entire UPS on Friday - guess we can all expect 
more outages then when they stuff that up.

D

On Mon, 13 Mar 2023 at 18:30, Chad Kelly 
mailto:c...@cpkws.com.au>> wrote:
Thought I would create a new thread on this, as the issue wasn’t related to 
AussieBB.
It is looking like this Melbourne outage is similar to the Sydney outage back 
in 2018, in that it appears to have been yet again caused by an issue with 
their UPS systems yet again.
I’ll leave this hear 
https://www.arnnet.com.au/article/633314/servers-australia-uproots-services-after-vocus-data-centre-outage/
 and this article on today’s outage as well.
https://www.itnews.com.au/news/vocus-melbourne-data-centre-hit-by-power-outage-591987

Regards Chad.



Chad Kelly
Manager
CPK Web Services
Phone 03 52730246
Web https://www.cpkws.com.au


___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog


--
veg·e·tar·i·an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog



___

AusNOG mailing list

AusNOG@lists.ausnog.net

https://lists.ausnog.net/mailman/listinfo/ausnog

--

** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL



/* Matt Perkins

Direct 02 8916 8101 Spectrum Networks Ptd. Ltd.

Office 1300 133 299 
m...@spectrum.com.au

ABN 66 090 112 913  Level 6, 350 George Street Sydney 2000



*/
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Vocus Datacentre outage.

[Matt Perkins]

SY1 had A or B fail a few years back if my memory serves me correctly.  
But I was referring  London and Manchester which both had total outages 
in recent years.


Matt.


On 14/3/2023 11:26 am, Nathan Brookfield wrote:


I’ve never had an A+B power rail failure at Equinix in 18 years I’ve 
been in there facilities, TOUCH WOOD!!!


*Nathan Brookfield | VK2NAB
*General Manager

*p*: 1300 592 330 | *m*: 0412 266 008
*e*: nathan.brookfi...@iperium.com.au | *w*: iperium.com.au

Suite 4.02, 189 Kent Street Sydney NSW 2000**


*Your Telco Team*

DISCLAIMER: This document is intended solely for the named addressee. 
This electronic communication, which includes any files or attachments 
thereto, contains proprietary or confidential information and may be 
privileged and otherwise protected under copyright or other applicable 
intellectual property laws. The use, disclosure, copying or 
distribution of any of the information contained in this document, by 
any person other than the addressee, is strictly prohibited. If you 
received this document in error, please contact the sender immediately 
and delete all the material from any computer. Confidentiality and 
legal privilege are not waived or lost by reason of mistaken delivery 
to you. Any views or opinions presented are solely those of the author 
and do not necessarily represent those of Iperium.


WARNING: Computer viruses can be transmitted via email. The recipient 
should check this email and any attachments for the presence of 
viruses. Iperium accepts no liability for any damage caused by any 
virus transmitted by this email.


*From:*AusNOG  *On Behalf Of *Matt 
Perkins

*Sent:* Tuesday, 14 March 2023 11:18 AM
*To:* ausnog@lists.ausnog.net
*Subject:* Re: [AusNOG] Vocus Datacentre outage.

Yea because Eqinix has never had a power outage. :)

I think it's a little premature to judge at the moment. Doesn't sound 
good based on the current info but we have all had bad days im sure.


Matt

On 14/3/2023 11:06 am, Bradley Amm wrote:

Friends like let friends put gear in a Vocus dtacenter. Only
NextDC or Equinix :)

Sent from my iPhone



On 14 Mar 2023, at 7:08 am, DaZZa 
 wrote:



Doesn't say a whole lot about Vocus' data center design
methodology, nor their periodic contingency testing.

They're apparently replacing the entire UPS on Friday - guess
we can all expect more outages then when they stuff that up.

D

On Mon, 13 Mar 2023 at 18:30, Chad Kelly 
wrote:

Thought I would create a new thread on this, as the issue
wasn’t related to AussieBB.

It is looking like this Melbourne outage is similar to the
Sydney outage back in 2018, in that it appears to have
been yet again caused by an issue with their UPS systems
yet again.

I’ll leave this hear

https://www.arnnet.com.au/article/633314/servers-australia-uproots-services-after-vocus-data-centre-outage/
and this article on today’s outage as well.


https://www.itnews.com.au/news/vocus-melbourne-data-centre-hit-by-power-outage-591987

Regards Chad.

Chad Kelly

Manager

CPK Web Services

Phone 03 52730246

Web https://www.cpkws.com.au



___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog



-- 


veg·e·tar·i·an:
Ancient tribal slang for the village idiot who can't hunt,
fish or ride

___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog



___

AusNOG mailing list

AusNOG@lists.ausnog.net

https://lists.ausnog.net/mailman/listinfo/ausnog

--
** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL
/* Matt Perkins
     Direct 02 8916 8101 Spectrum Networks Ptd. Ltd.
     Office 1300 133 299m...@spectrum.com.au
     ABN 66 090 112 913  Level 6, 350 George Street Sydney 2000
*/


--
** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL

/* Matt Perkins
Direct 02 8916 8101 Spectrum Networks Ptd. Ltd.
Office 1300 133 299m...@spectrum.com.au
ABN 66 090 112 913  Level 6, 350 George Street Sydney 2000

*/
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Vocus Datacentre outage.

[Nathan Brookfield]

Have 100% had A and B fail separately at SY1 but never both, thankfully!!!

Could have been part of the facility though so don’t quote me.

UK on the other hand, fail!

From: Matt Perkins 
Sent: Tuesday, 14 March 2023 11:30 AM
To: Nathan Brookfield ; 
ausnog@lists.ausnog.net
Subject: Re: [AusNOG] Vocus Datacentre outage.


SY1 had A or B fail a few years back if my memory serves me correctly.  But I 
was referring  London and Manchester which both had total outages in recent 
years.

Matt.


On 14/3/2023 11:26 am, Nathan Brookfield wrote:
I’ve never had an A+B power rail failure at Equinix in 18 years I’ve been in 
there facilities, TOUCH WOOD!!!

Nathan Brookfield | VK2NAB
General Manager

p: 1300 592 330 | m: 0412 266 008
e: nathan.brookfi...@iperium.com.au | 
w: iperium.com.au

Suite 4.02, 189 Kent Street Sydney NSW 2000
[cid:image001.png@01D95669.B5A95AB0]
Your Telco Team
DISCLAIMER: This document is intended solely for the named addressee. This 
electronic communication, which includes any files or attachments thereto, 
contains proprietary or confidential information and may be privileged and 
otherwise protected under copyright or other applicable intellectual property 
laws. The use, disclosure, copying or distribution of any of the information 
contained in this document, by any person other than the addressee, is strictly 
prohibited. If you received this document in error, please contact the sender 
immediately and delete all the material from any computer. Confidentiality and 
legal privilege are not waived or lost by reason of mistaken delivery to you. 
Any views or opinions presented are solely those of the author and do not 
necessarily represent those of Iperium.

WARNING: Computer viruses can be transmitted via email. The recipient should 
check this email and any attachments for the presence of viruses. Iperium 
accepts no liability for any damage caused by any virus transmitted by this 
email.
From: AusNOG 
 On 
Behalf Of Matt Perkins
Sent: Tuesday, 14 March 2023 11:18 AM
To: ausnog@lists.ausnog.net
Subject: Re: [AusNOG] Vocus Datacentre outage.


Yea because Eqinix has never had a power outage. :)

I think it's a little premature to judge at the moment. Doesn't sound good 
based on the current info but we have all had bad days im sure.

Matt


On 14/3/2023 11:06 am, Bradley Amm wrote:
Friends like let friends put gear in a Vocus dtacenter. Only NextDC or Equinix 
:)
Sent from my iPhone



On 14 Mar 2023, at 7:08 am, DaZZa 
 wrote:

Doesn't say a whole lot about Vocus' data center design methodology, nor their 
periodic contingency testing.

They're apparently replacing the entire UPS on Friday - guess we can all expect 
more outages then when they stuff that up.

D

On Mon, 13 Mar 2023 at 18:30, Chad Kelly 
mailto:c...@cpkws.com.au>> wrote:
Thought I would create a new thread on this, as the issue wasn’t related to 
AussieBB.
It is looking like this Melbourne outage is similar to the Sydney outage back 
in 2018, in that it appears to have been yet again caused by an issue with 
their UPS systems yet again.
I’ll leave this hear 
https://www.arnnet.com.au/article/633314/servers-australia-uproots-services-after-vocus-data-centre-outage/
 and this article on today’s outage as well.
https://www.itnews.com.au/news/vocus-melbourne-data-centre-hit-by-power-outage-591987

Regards Chad.



Chad Kelly
Manager
CPK Web Services
Phone 03 52730246
Web https://www.cpkws.com.au


___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog


--
veg·e·tar·i·an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog




___

AusNOG mailing list

AusNOG@lists.ausnog.net

https://lists.ausnog.net/mailman/listinfo/ausnog

--

** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL



/* Matt Perkins

Direct 02 8916 8101 Spectrum Networks Ptd. Ltd.

Office 1300 133 299 
m...@spectrum.com.au

ABN 66 090 112 913  Level 6, 350 George Street Sydney 2000



*/

--

** DO NOT SEND PRIVATE OR CONFIDENTIAL MATERIAL VIA EMAIL



/* Matt Perkins

Direct 02 8916 8101 Spectrum Networks Ptd. Ltd.

Office 1300 133 299 
m...@spectrum.com.au

ABN 66 090 112 913  Level 6, 350 George Street Sydney 2000



*/
___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

[AusNOG] ISIF Asia 2023 grant applications now open

[Dave Dawson]

Hi all -


Just letting you know that the ISIF Asia grants are open. We'd love to see 
applications from AusNOG members. Applications close April 30. Info below.

I am happy to field queries, but also check the links below and our FAQs as 
they might have covered it: https://isif.asia/faq/

Thanks,
David Dawson
(APNIC Foundation comms)
--

ISIF Asia is opening its 2023 grant cycle for applications.

ISIF Asia Grants are focused on network operations and Internet development in 
the Asia Pacific. Almost any kind of organization can apply. Please share the 
news with your networks, we have a variety of different grants on offer.

The application link: https://isif.asia/apply/

Grants are across three broad areas:

Inclusion: Initiatives that help ensure everyone has meaningful access to the 
Internet, online applications and services.

Infrastructure: Initiatives to increase Internet speed, reduce maintenance and 
operational costs, and improve reliability and/or security.

Knowledge: Initiatives that develop technical capacity and/or research around 
Internet network operations for technical and non-technical audiences.

Within these three program areas, there are additional types of grant:

Ian Peter Grants for Internet and the Environment

The Ian Peter Grants for Internet and the Environment can be across any of the 
three program areas and are to support innovative solutions to the 
environmental impact of the Internet. They commemorate the life and work of Ian 
Peter, Internet pioneer and environmental activist.

IPv6 Deployment Grants

ISIF Asia's IPv6 Deployment Grants are also open for organizations planning to 
deploy IPv6 in the Asia Pacific. IPv6 Deployment Grants range between USD 
30,000 and USD 250,000. These are open all year round and are part of the 
Infrastructure program.

Get in your applications now! Applications (excluding the IPv6 Deployment 
grants) close 30 April 2023, 11:59 AM (UTC+10).

More information is available on the ISIF Asia blog.

https://isif.asia/2023-isif-asia-grants-now-open-for-applications/


___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog

Re: [AusNOG] Vocus Datacentre outage.

[Christopher Hawker]

SY1 apparently experienced a full power outage back in July 2009, according to 
AusNOG Archives. Some reported full outages, some partial. Was not a DC fault 
though, looks like a much larger grid issue where a fault at the Bayswater 
Power Station in the Hunter Valley (which affected generators in QLD and VIC) 
caused about 1000MW to be dropped from the grid, resulting in rolling blackouts 
across QLD, NSW, TAS, SA and VIC.

https://lists.ausnog.net/pipermail/ausnog/2009-July/003222.html

CH.

From: AusNOG  on behalf of Nathan Brookfield 

Sent: Tuesday, March 14, 2023 11:40 AM
To: Matt Perkins ; ausnog@lists.ausnog.net 

Subject: Re: [AusNOG] Vocus Datacentre outage.


Have 100% had A and B fail separately at SY1 but never both, thankfully!!!



Could have been part of the facility though so don’t quote me.



UK on the other hand, fail!



From: Matt Perkins 
Sent: Tuesday, 14 March 2023 11:30 AM
To: Nathan Brookfield ; 
ausnog@lists.ausnog.net
Subject: Re: [AusNOG] Vocus Datacentre outage.



SY1 had A or B fail a few years back if my memory serves me correctly.  But I 
was referring  London and Manchester which both had total outages in recent 
years.

Matt.



On 14/3/2023 11:26 am, Nathan Brookfield wrote:

I’ve never had an A+B power rail failure at Equinix in 18 years I’ve been in 
there facilities, TOUCH WOOD!!!



Nathan Brookfield | VK2NAB
General Manager

p: 1300 592 330 | m: 0412 266 008
e: nathan.brookfi...@iperium.com.au | 
w: iperium.com.au

Suite 4.02, 189 Kent Street Sydney NSW 2000

[cid:image001.png@01D95669.B5A95AB0]
Your Telco Team

DISCLAIMER: This document is intended solely for the named addressee. This 
electronic communication, which includes any files or attachments thereto, 
contains proprietary or confidential information and may be privileged and 
otherwise protected under copyright or other applicable intellectual property 
laws. The use, disclosure, copying or distribution of any of the information 
contained in this document, by any person other than the addressee, is strictly 
prohibited. If you received this document in error, please contact the sender 
immediately and delete all the material from any computer. Confidentiality and 
legal privilege are not waived or lost by reason of mistaken delivery to you. 
Any views or opinions presented are solely those of the author and do not 
necessarily represent those of Iperium.

WARNING: Computer viruses can be transmitted via email. The recipient should 
check this email and any attachments for the presence of viruses. Iperium 
accepts no liability for any damage caused by any virus transmitted by this 
email.

From: AusNOG 
 On 
Behalf Of Matt Perkins
Sent: Tuesday, 14 March 2023 11:18 AM
To: ausnog@lists.ausnog.net
Subject: Re: [AusNOG] Vocus Datacentre outage.



Yea because Eqinix has never had a power outage. :)

I think it's a little premature to judge at the moment. Doesn't sound good 
based on the current info but we have all had bad days im sure.

Matt



On 14/3/2023 11:06 am, Bradley Amm wrote:

Friends like let friends put gear in a Vocus dtacenter. Only NextDC or Equinix 
:)

Sent from my iPhone




On 14 Mar 2023, at 7:08 am, DaZZa 
 wrote:



Doesn't say a whole lot about Vocus' data center design methodology, nor their 
periodic contingency testing.



They're apparently replacing the entire UPS on Friday - guess we can all expect 
more outages then when they stuff that up.



D



On Mon, 13 Mar 2023 at 18:30, Chad Kelly 
mailto:c...@cpkws.com.au>> wrote:

Thought I would create a new thread on this, as the issue wasn’t related to 
AussieBB.

It is looking like this Melbourne outage is similar to the Sydney outage back 
in 2018, in that it appears to have been yet again caused by an issue with 
their UPS systems yet again.

I’ll leave this hear 
https://www.arnnet.com.au/article/633314/servers-australia-uproots-services-after-vocus-data-centre-outage/
 and this article on today’s outage as well.

https://www.itnews.com.au/news/vocus-melbourne-data-centre-hit-by-power-outage-591987



Regards Chad.







Chad Kelly

Manager

CPK Web Services

Phone 03 52730246

Web https://www.cpkws.com.au





___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog


--

veg·e·tar·i·an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride

___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog




___

AusNOG mailing list

AusNOG@lists.ausnog.net

https://lists.ausnog.net/mailman/listinfo/ausnog

--

** DO NOT SEND PRIVATE OR CO

Re: [AusNOG] Vocus Datacentre outage.

[Nathan Brookfield]

I knew I didn’t remember it for a reason 😂


[AusNOG] Equinix 
Outage
lists.ausnog.net
[X]

On 14 Mar 2023, at 10:50, Christopher Hawker  wrote:


SY1 apparently experienced a full power outage back in July 2009, according to 
AusNOG Archives. Some reported full outages, some partial. Was not a DC fault 
though, looks like a much larger grid issue where a fault at the Bayswater 
Power Station in the Hunter Valley (which affected generators in QLD and VIC) 
caused about 1000MW to be dropped from the grid, resulting in rolling blackouts 
across QLD, NSW, TAS, SA and VIC.

https://lists.ausnog.net/pipermail/ausnog/2009-July/003222.html

CH.

From: AusNOG  on behalf of Nathan Brookfield 

Sent: Tuesday, March 14, 2023 11:40 AM
To: Matt Perkins ; ausnog@lists.ausnog.net 

Subject: Re: [AusNOG] Vocus Datacentre outage.


Have 100% had A and B fail separately at SY1 but never both, thankfully!!!



Could have been part of the facility though so don’t quote me.



UK on the other hand, fail!



From: Matt Perkins 
Sent: Tuesday, 14 March 2023 11:30 AM
To: Nathan Brookfield ; 
ausnog@lists.ausnog.net
Subject: Re: [AusNOG] Vocus Datacentre outage.



SY1 had A or B fail a few years back if my memory serves me correctly.  But I 
was referring  London and Manchester which both had total outages in recent 
years.

Matt.



On 14/3/2023 11:26 am, Nathan Brookfield wrote:

I’ve never had an A+B power rail failure at Equinix in 18 years I’ve been in 
there facilities, TOUCH WOOD!!!



Nathan Brookfield | VK2NAB
General Manager

p: 1300 592 330 | m: 0412 266 008
e: nathan.brookfi...@iperium.com.au | 
w: iperium.com.au

Suite 4.02, 189 Kent Street Sydney NSW 2000



Your Telco Team

DISCLAIMER: This document is intended solely for the named addressee. This 
electronic communication, which includes any files or attachments thereto, 
contains proprietary or confidential information and may be privileged and 
otherwise protected under copyright or other applicable intellectual property 
laws. The use, disclosure, copying or distribution of any of the information 
contained in this document, by any person other than the addressee, is strictly 
prohibited. If you received this document in error, please contact the sender 
immediately and delete all the material from any computer. Confidentiality and 
legal privilege are not waived or lost by reason of mistaken delivery to you. 
Any views or opinions presented are solely those of the author and do not 
necessarily represent those of Iperium.

WARNING: Computer viruses can be transmitted via email. The recipient should 
check this email and any attachments for the presence of viruses. Iperium 
accepts no liability for any damage caused by any virus transmitted by this 
email.

From: AusNOG 
 On 
Behalf Of Matt Perkins
Sent: Tuesday, 14 March 2023 11:18 AM
To: ausnog@lists.ausnog.net
Subject: Re: [AusNOG] Vocus Datacentre outage.



Yea because Eqinix has never had a power outage. :)

I think it's a little premature to judge at the moment. Doesn't sound good 
based on the current info but we have all had bad days im sure.

Matt



On 14/3/2023 11:06 am, Bradley Amm wrote:

Friends like let friends put gear in a Vocus dtacenter. Only NextDC or Equinix 
:)

Sent from my iPhone




On 14 Mar 2023, at 7:08 am, DaZZa 
 wrote:



Doesn't say a whole lot about Vocus' data center design methodology, nor their 
periodic contingency testing.



They're apparently replacing the entire UPS on Friday - guess we can all expect 
more outages then when they stuff that up.



D



On Mon, 13 Mar 2023 at 18:30, Chad Kelly 
mailto:c...@cpkws.com.au>> wrote:

Thought I would create a new thread on this, as the issue wasn’t related to 
AussieBB.

It is looking like this Melbourne outage is similar to the Sydney outage back 
in 2018, in that it appears to have been yet again caused by an issue with 
their UPS systems yet again.

I’ll leave this hear 
https://www.arnnet.com.au/article/633314/servers-australia-uproots-services-after-vocus-data-centre-outage/
 and this article on today’s outage as well.

https://www.itnews.com.au/news/vocus-melbourne-data-centre-hit-by-power-outage-591987



Regards Chad.







Chad Kelly

Manager

CPK Web Services

Phone 03 52730246

Web https://www.cpkws.com.au





___
AusNOG mailing list
AusNOG@lists.ausnog.net
https://lists.ausnog.net/mailman/listinfo/ausnog


--

veg·e·tar·i·an:
Ancient tribal slang for the village idiot who can't hunt, fish or ride

_

Re: [AusNOG] Vocus Datacentre outage.

[Matt Perkins]

The questions left to be answered for Vocus is why did it take 12 hours 
to restore one rail. Where was the static bypass and failing that where 
was the maintenance bypass.  Im guessing that's what it's on maintenance 
bypass now and the slow action may have something to do with the public 
holiday in Vic..



I shouldn't speculate. But it is fun.


 Something seriously let go in that UPS. If i had to put my electrical 
engineers hat on I would say their load capacitor maintenance was not 
done and let go. Or someone was working upstream and broke the 
natural/earth bond. But there's a lot of parts in a system like that , 
most modern DC UPS setups are fairly robust.  I once had 3 strings of 
batteries catch fire and melt down and the UPS didn't miss a beat other 
then spewing a lot of alarms. We moved strings around on the fly and 
nobody even noticed.



The other things that seems odd is why did an A failure trip B. Sounds 
like a single supply is at surge/break capacity. Also a worry.



But in the end we all have redundancy around 530 Collins right ? Right ?


Matt



On 14/3/2023 11:50 am, Christopher Hawker wrote:
SY1 apparently experienced a full power outage back in July 2009, 
according to AusNOG Archives. Some reported full outages, some 
partial. Was not a DC fault though, looks like a much larger grid 
issue where a fault at the Bayswater Power Station in the Hunter 
Valley (which affected generators in QLD and VIC) caused about 1000MW 
to be dropped from the grid, resulting in rolling blackouts across 
QLD, NSW, TAS, SA and VIC.


https://lists.ausnog.net/pipermail/ausnog/2009-July/003222.html

CH.

*From:* AusNOG  on behalf of Nathan 
Brookfield 

*Sent:* Tuesday, March 14, 2023 11:40 AM
*To:* Matt Perkins ; ausnog@lists.ausnog.net 


*Subject:* Re: [AusNOG] Vocus Datacentre outage.

Have 100% had A and B fail separately at SY1 but never both, thankfully!!!

Could have been part of the facility though so don’t quote me.

UK on the other hand, fail!

*From:*Matt Perkins 
*Sent:* Tuesday, 14 March 2023 11:30 AM
*To:* Nathan Brookfield ; 
ausnog@lists.ausnog.net

*Subject:* Re: [AusNOG] Vocus Datacentre outage.

SY1 had A or B fail a few years back if my memory serves me 
correctly.  But I was referring  London and Manchester which both had 
total outages in recent years.


Matt.

On 14/3/2023 11:26 am, Nathan Brookfield wrote:

I’ve never had an A+B power rail failure at Equinix in 18 years
I’ve been in there facilities, TOUCH WOOD!!!

*Nathan Brookfield | VK2NAB
*General Manager

*p*: 1300 592 330 | *m*: 0412 266 008
*e*: nathan.brookfi...@iperium.com.au | *w*: iperium.com.au

Suite 4.02, 189 Kent Street Sydney NSW 2000


*Your Telco Team*

DISCLAIMER: This document is intended solely for the named
addressee. This electronic communication, which includes any files
or attachments thereto, contains proprietary or confidential
information and may be privileged and otherwise protected under
copyright or other applicable intellectual property laws. The use,
disclosure, copying or distribution of any of the information
contained in this document, by any person other than the
addressee, is strictly prohibited. If you received this document
in error, please contact the sender immediately and delete all the
material from any computer. Confidentiality and legal privilege
are not waived or lost by reason of mistaken delivery to you. Any
views or opinions presented are solely those of the author and do
not necessarily represent those of Iperium.

WARNING: Computer viruses can be transmitted via email. The
recipient should check this email and any attachments for the
presence of viruses. Iperium accepts no liability for any damage
caused by any virus transmitted by this email.

*From:*AusNOG 
 *On Behalf Of *Matt Perkins
*Sent:* Tuesday, 14 March 2023 11:18 AM
*To:* ausnog@lists.ausnog.net
*Subject:* Re: [AusNOG] Vocus Datacentre outage.

Yea because Eqinix has never had a power outage. :)

I think it's a little premature to judge at the moment. Doesn't
sound good based on the current info but we have all had bad days
im sure.

Matt

On 14/3/2023 11:06 am, Bradley Amm wrote:

Friends like let friends put gear in a Vocus dtacenter. Only
NextDC or Equinix :)

Sent from my iPhone




On 14 Mar 2023, at 7:08 am, DaZZa 
 wrote:



Doesn't say a whole lot about Vocus' data center design
methodology, nor their periodic contingency testing.

They're apparently replacing the entire UPS on Friday -
guess we can all expect more outages then when they stuff
that up.

D

On Mon, 13 Mar 2023 at 1

Re: [AusNOG] Vocus Datacentre outage.

[Joel Nath]

In EQX defence Sy1 didn’t go dark… it was multiple (3-5) rows of rack that lost 
power in 2009.

We were one of the customers heavily impacted in 2009 but understood the reason 
why it happened and EQX fixed it all up.

Vocus Doody st different story and now melb

We pulled out of Doody after the drama we faced there.

Joel.

From: AusNOG  On Behalf Of Nathan Brookfield
Sent: Tuesday, 14 March 2023 11:54 AM
To: Christopher Hawker 
Cc: ausnog@lists.ausnog.net
Subject: Re: [AusNOG] Vocus Datacentre outage.

I knew I didn’t remember it for a reason 😂

[AusNOG] Equinix 
Outage
lists.ausnog.net

[X]



On 14 Mar 2023, at 10:50, Christopher Hawker 
mailto:ch...@thesysadmin.dev>> wrote:


SY1 apparently experienced a full power outage back in July 2009, according to 
AusNOG Archives. Some reported full outages, some partial. Was not a DC fault 
though, looks like a much larger grid issue where a fault at the Bayswater 
Power Station in the Hunter Valley (which affected generators in QLD and VIC) 
caused about 1000MW to be dropped from the grid, resulting in rolling blackouts 
across QLD, NSW, TAS, SA and VIC.

https://lists.ausnog.net/pipermail/ausnog/2009-July/003222.html

CH.

From: AusNOG 
mailto:ausnog-boun...@lists.ausnog.net>> on 
behalf of Nathan Brookfield 
mailto:nathan.brookfi...@iperium.com.au>>
Sent: Tuesday, March 14, 2023 11:40 AM
To: Matt Perkins mailto:m...@spectrum.com.au>>; 
ausnog@lists.ausnog.net 
mailto:ausnog@lists.ausnog.net>>
Subject: Re: [AusNOG] Vocus Datacentre outage.


Have 100% had A and B fail separately at SY1 but never both, thankfully!!!



Could have been part of the facility though so don’t quote me.



UK on the other hand, fail!



From: Matt Perkins mailto:m...@spectrum.com.au>>
Sent: Tuesday, 14 March 2023 11:30 AM
To: Nathan Brookfield 
mailto:nathan.brookfi...@iperium.com.au>>; 
ausnog@lists.ausnog.net
Subject: Re: [AusNOG] Vocus Datacentre outage.



SY1 had A or B fail a few years back if my memory serves me correctly.  But I 
was referring  London and Manchester which both had total outages in recent 
years.

Matt.



On 14/3/2023 11:26 am, Nathan Brookfield wrote:

I’ve never had an A+B power rail failure at Equinix in 18 years I’ve been in 
there facilities, TOUCH WOOD!!!



Nathan Brookfield | VK2NAB
General Manager

p: 1300 592 330 | m: 0412 266 008
e: nathan.brookfi...@iperium.com.au | 
w: iperium.com.au

Suite 4.02, 189 Kent Street Sydney NSW 2000


Your Telco Team

DISCLAIMER: This document is intended solely for the named addressee. This 
electronic communication, which includes any files or attachments thereto, 
contains proprietary or confidential information and may be privileged and 
otherwise protected under copyright or other applicable intellectual property 
laws. The use, disclosure, copying or distribution of any of the information 
contained in this document, by any person other than the addressee, is strictly 
prohibited. If you received this document in error, please contact the sender 
immediately and delete all the material from any computer. Confidentiality and 
legal privilege are not waived or lost by reason of mistaken delivery to you. 
Any views or opinions presented are solely those of the author and do not 
necessarily represent those of Iperium.

WARNING: Computer viruses can be transmitted via email. The recipient should 
check this email and any attachments for the presence of viruses. Iperium 
accepts no liability for any damage caused by any virus transmitted by this 
email.

From: AusNOG 
 On 
Behalf Of Matt Perkins
Sent: Tuesday, 14 March 2023 11:18 AM
To: ausnog@lists.ausnog.net
Subject: Re: [AusNOG] Vocus Datacentre outage.



Yea because Eqinix has never had a power outage. :)

I think it's a little premature to judge at the moment. Doesn't sound good 
based on the current info but we have all had bad days im sure.

Matt



On 14/3/2023 11:06 am, Bradley Amm wrote:

Friends like let friends put gear in a Vocus dtacenter. Only NextDC or Equinix 
:)

Sent from my iPhone



On 14 Mar 2023, at 7:08 am, DaZZa 
 wrote:



Doesn't say a whole lot about Vocus' data center design methodology, nor their 
periodic contingency testing.



They're apparently replacing the entire UPS on Friday - guess we can all expect 
more outages then when they stuff that up.



D



On Mon, 13 Mar 2023 at 18:30, Chad Kelly 
mailto:c...@cpkws.com.au>> wrote:

Thought I would create a new thread on this, as the issue wasn’t related to 
AussieBB.

It is looking like this Melbourne outage is similar to the Sydney outage back 
in 2018,

Re: [AusNOG] Vocus Datacentre outage.

[John Edwards]

Don't rule out a firmware setting that lay dormant or was introduced
through overzealous or simultaneous patching in a maintenance window.

This was how SA ended up with a total blackout a few years back - a default
setting in a particular model of wind turbine shut it down in response to a
previously unseen power anomaly - caused by a power outage elsewhere on the
grid.

John


On Tue, 14 Mar 2023 at 11:48, Matt Perkins  wrote:

> The questions left to be answered for Vocus is why did it take 12 hours to
> restore one rail. Where was the static bypass and failing that where was
> the maintenance bypass.  Im guessing that's what it's on maintenance bypass
> now and the slow action may have something to do with the public holiday in
> Vic..
>
>
> I shouldn't speculate. But it is fun.
>
>
>  Something seriously let go in that UPS. If i had to put my electrical
> engineers hat on I would say their load capacitor maintenance was not done
> and let go. Or someone was working upstream and broke the natural/earth
> bond. But there's a lot of parts in a system like that , most modern DC UPS
> setups are fairly robust.  I once had 3 strings of batteries catch fire and
> melt down and the UPS didn't miss a beat other then spewing a lot of
> alarms. We moved strings around on the fly and nobody even noticed.
>
>
> The other things that seems odd is why did an A failure trip B.  Sounds
> like a single supply is at surge/break capacity. Also a worry.
>
>
> But in the end we all have redundancy around 530 Collins right ? Right ?
>
>
> Matt
>
>
>
> On 14/3/2023 11:50 am, Christopher Hawker wrote:
>
> SY1 apparently experienced a full power outage back in July 2009,
> according to AusNOG Archives. Some reported full outages, some partial. Was
> not a DC fault though, looks like a much larger grid issue where a fault at
> the Bayswater Power Station in the Hunter Valley (which affected generators
> in QLD and VIC) caused about 1000MW to be dropped from the grid, resulting
> in rolling blackouts across QLD, NSW, TAS, SA and VIC.
>
> https://lists.ausnog.net/pipermail/ausnog/2009-July/003222.html
>
> CH.
> --
> *From:* AusNOG 
>  on behalf of Nathan Brookfield
>  
> *Sent:* Tuesday, March 14, 2023 11:40 AM
> *To:* Matt Perkins  ;
> ausnog@lists.ausnog.net 
> 
> *Subject:* Re: [AusNOG] Vocus Datacentre outage.
>
>
> Have 100% had A and B fail separately at SY1 but never both, thankfully!!!
>
>
>
> Could have been part of the facility though so don’t quote me.
>
>
>
> UK on the other hand, fail!
>
>
>
> *From:* Matt Perkins  
> *Sent:* Tuesday, 14 March 2023 11:30 AM
> *To:* Nathan Brookfield 
> ; ausnog@lists.ausnog.net
> *Subject:* Re: [AusNOG] Vocus Datacentre outage.
>
>
>
> SY1 had A or B fail a few years back if my memory serves me correctly.
> But I was referring  London and Manchester which both had total outages in
> recent years.
>
> Matt.
>
>
>
> On 14/3/2023 11:26 am, Nathan Brookfield wrote:
>
> I’ve never had an A+B power rail failure at Equinix in 18 years I’ve been
> in there facilities, TOUCH WOOD!!!
>
>
>
>
> *Nathan Brookfield | VK2NAB *General Manager
>
> *p*: 1300 592 330 | *m*: 0412 266 008
> *e*: nathan.brookfi...@iperium.com.au | *w*: iperium.com.au
>
> Suite 4.02, 189 Kent Street Sydney NSW 2000
>
>
> *Your Telco Team*
>
> DISCLAIMER: This document is intended solely for the named addressee. This
> electronic communication, which includes any files or attachments thereto,
> contains proprietary or confidential information and may be privileged and
> otherwise protected under copyright or other applicable intellectual
> property laws. The use, disclosure, copying or distribution of any of the
> information contained in this document, by any person other than the
> addressee, is strictly prohibited. If you received this document in error,
> please contact the sender immediately and delete all the material from any
> computer. Confidentiality and legal privilege are not waived or lost by
> reason of mistaken delivery to you. Any views or opinions presented are
> solely those of the author and do not necessarily represent those of
> Iperium.
>
> WARNING: Computer viruses can be transmitted via email. The recipient
> should check this email and any attachments for the presence of viruses.
> Iperium accepts no liability for any damage caused by any virus transmitted
> by this email.
>
> *From:* AusNOG 
>  *On Behalf Of *Matt Perkins
> *Sent:* Tuesday, 14 March 2023 11:18 AM
> *To:* ausnog@lists.ausnog.net
> *Subject:* Re: [AusNOG] Vocus Datacentre outage.
>
>
>
> Yea because Eqinix has never had a power outage. :)
>
> I think it's a little premature to judge at the moment. Doesn't sound good
> based on the current info but we have all had bad days im sure.
>
> Matt
>
>
>
> On 14/3/2023 11:06 am, Bradley Amm wrote:
>
> Friends like let friends put gear in a Vocus dtacenter. Only NextDC or
> Equinix :)
>
> Sent from my iPhone
>
>
>
>
> On 14 Mar 2023, at 7:08 am, DaZZa 
>  wrote:
>

Re: [AusNOG] Vocus Datacentre outage.

[Mark Dignam]

Something that I have to keep reminding beancounters with A/B planning is don’t 
go above 40% load on either side.

 

Its very difficult to convince a CFO who’s entire life is spent with a 
spreadsheet open in front of him is that N+1 means typically Double, and that 
Kw != Kva.

 

 

From: AusNOG  On Behalf Of Matt Perkins
Sent: Tuesday, March 14, 2023 9:18 AM
To: Christopher Hawker ; Nathan Brookfield 
; ausnog@lists.ausnog.net
Subject: Re: [AusNOG] Vocus Datacentre outage.

 

The questions left to be answered for Vocus is why did it take 12 hours to 
restore one rail. Where was the static bypass and failing that where was the 
maintenance bypass.  Im guessing that's what it's on maintenance bypass now and 
the slow action may have something to do with the public holiday in Vic..  

 

I shouldn't speculate. But it is fun. 

 

 Something seriously let go in that UPS. If i had to put my electrical 
engineers hat on I would say their load capacitor maintenance was not done and 
let go. Or someone was working upstream and broke the natural/earth bond. But 
there's a lot of parts in a system like that , most modern DC UPS setups are 
fairly robust.  I once had 3 strings of batteries catch fire and melt down and 
the UPS didn't miss a beat other then spewing a lot of alarms. We moved strings 
around on the fly and nobody even noticed.   

 

The other things that seems odd is why did an A failure trip B.  Sounds like a 
single supply is at surge/break capacity. Also a worry.  

 

But in the end we all have redundancy around 530 Collins right ? Right ? 

 

Matt

 

 

On 14/3/2023 11:50 am, Christopher Hawker wrote:

SY1 apparently experienced a full power outage back in July 2009, according to 
AusNOG Archives. Some reported full outages, some partial. Was not a DC fault 
though, looks like a much larger grid issue where a fault at the Bayswater 
Power Station in the Hunter Valley (which affected generators in QLD and VIC) 
caused about 1000MW to be dropped from the grid, resulting in rolling blackouts 
across QLD, NSW, TAS, SA and VIC.

 

https://lists.ausnog.net/pipermail/ausnog/2009-July/003222.html

 

CH.

  _  

From: AusNOG   
 on behalf of Nathan Brookfield  
 
Sent: Tuesday, March 14, 2023 11:40 AM
To: Matt Perkins   ; 
ausnog@lists.ausnog.net    
 
Subject: Re: [AusNOG] Vocus Datacentre outage. 

 

Have 100% had A and B fail separately at SY1 but never both, thankfully!!!

 

Could have been part of the facility though so don’t quote me.

 

UK on the other hand, fail!

 

From: Matt Perkins    
Sent: Tuesday, 14 March 2023 11:30 AM
To: Nathan Brookfield   
; ausnog@lists.ausnog.net 
 
Subject: Re: [AusNOG] Vocus Datacentre outage.

 

SY1 had A or B fail a few years back if my memory serves me correctly.  But I 
was referring  London and Manchester which both had total outages in recent 
years. 

Matt.

 

On 14/3/2023 11:26 am, Nathan Brookfield wrote:

I’ve never had an A+B power rail failure at Equinix in 18 years I’ve been in 
there facilities, TOUCH WOOD!!!

 

Nathan Brookfield | VK2NAB
General Manager

p: 1300 592 330 | m: 0412 266 008
e: nathan.brookfi...@iperium.com.au   
| w: iperium.com.au

Suite 4.02, 189 Kent Street Sydney NSW 2000


Your Telco Team

DISCLAIMER: This document is intended solely for the named addressee. This 
electronic communication, which includes any files or attachments thereto, 
contains proprietary or confidential information and may be privileged and 
otherwise protected under copyright or other applicable intellectual property 
laws. The use, disclosure, copying or distribution of any of the information 
contained in this document, by any person other than the addressee, is strictly 
prohibited. If you received this document in error, please contact the sender 
immediately and delete all the material from any computer. Confidentiality and 
legal privilege are not waived or lost by reason of mistaken delivery to you. 
Any views or opinions presented are solely those of the author and do not 
necessarily represent those of Iperium. 

WARNING: Computer viruses can be transmitted via email. The recipient should 
check this email and any attachments for the presence of viruses. Iperium 
accepts no liability for any damage caused by any virus transmitted by this 
email.

From: AusNOG   
 On Behalf Of Matt Perkins
Sent: Tuesday, 14 March 2023 11:18 AM
To: ausnog@lists.ausnog.net  
Subject: Re: [AusNOG] Vocus Datacentre outage.

 

Yea because Eqinix has never had a power outage. :) 

I think it's a little premature to judge at the moment. Doesn't sound good 
based on the curr

Re: [AusNOG] Vocus Datacentre outage.

[Jared Hirst | Servers Australia]

Good times back then Joel! It was a UPS issue back there too if I remember, and 
the A and B feeds were all fed into the one UPS….


A tonne of people shifted within that week, for me it was the way that it was 
palmed off, as you say, others might have issues but they own those issues and 
rectify them…



Jared Hirst
Chief Executive Officer
P: (02) 8115 8801 
4 Amy Close, Wyong, NSW 2259
Need assistance? We are here 24/7 +61 2 8115  

--- original message ---
On March 14, 2023 at 12:26 PM GMT+11 j...@zonenetworks.com.au wrote:



In EQX defence Sy1 didn’t go dark… it was multiple (3-5) rows of rack that lost 
power in 2009.



We were one of the customers heavily impacted in 2009 but understood the reason 
why it happened and EQX fixed it all up.



Vocus Doody st different story and now melb



We pulled out of Doody after the drama we faced there.



Joel.



From: AusNOG 
On Behalf Of Nathan Brookfield

Sent: Tuesday, 14 March 2023 11:54 AM

To: Christopher Hawker 

Cc: ausnog@lists.ausnog.net

Subject: Re: [AusNOG] Vocus Datacentre outage.





I knew I didn’t remember it for a reason 
😂 




[AusNOG] Equinix Outage

lists.ausnog.net











On 14 Mar 2023, at 10:50, Christopher Hawker  wrote:






SY1 apparently experienced a full power outage back in July 2009, according to 
AusNOG Archives. Some reported full outages, some partial. Was not a DC fault
though, looks like a much larger grid issue where a fault at the Bayswater 
Power Station in the Hunter Valley (which affected generators in QLD and VIC) 
caused about 1000MW to be dropped from the grid, resulting in rolling blackouts 
across QLD, NSW, TAS, SA
and VIC.





https://lists.ausnog.net/pipermail/ausnog/2009-July/003222.html





CH.




From: AusNOG 
on behalf of Nathan Brookfield 

Sent: Tuesday, March 14, 2023 11:40 AM

To: Matt Perkins ;
ausnog@lists.ausnog.net 

Subject: Re: [AusNOG] Vocus Datacentre outage. 





Have 100% had A and B fail separately at SY1 but never both, thankfully!!!



Could have been part of the facility though so don’t quote me.



UK on the other hand, fail!



From: Matt Perkins 

Sent: Tuesday, 14 March 2023 11:30 AM

To: Nathan Brookfield ;
ausnog@lists.ausnog.net

Subject: Re: [AusNOG] Vocus Datacentre outage.





SY1 had A or B fail a few years back if my memory serves me correctly. But I 
was referring London and Manchester which both had total outages in recent 
years.

Matt.



On 14/3/2023 11:26 am, Nathan Brookfield wrote:


> I’ve never had an A+B power rail failure at Equinix in 18 years I’ve been in 
> there facilities, TOUCH WOOD!!!

> 

> Nathan Brookfield | VK2NAB

> General Manager


> p: 1300 592 330 |>m: 0412 266 008

> e:>nathan.brookfi...@iperium.com.au | w: iperium.com.au


> Suite 4.02, 189 Kent Street Sydney NSW 2000

> 



> Your Telco Team


> DISCLAIMER: This document is intended solely for the named addressee. This 
> electronic communication, which includes any files or attachments thereto, 
> contains proprietary> or confidential information and may be privileged and 
> otherwise protected under copyright or other applicable intellectual property 
> laws. The use, disclosure, copying or distribution of any of the information 
> contained in this document, by any person other> than the addressee, is 
> strictly prohibited. If you received this document in error, please contact 
> the sender immediately and delete all the material from any computer. 
> Confidentiality and legal privilege are not waived or lost by reason of 
> mistaken delivery> to you. Any views or opinions presented are solely those 
> of the author and do not necessarily represent those of Iperium. 


> WARNING: Computer viruses can be transmitted via email. The recipient should 
> check this email and any attachments for the presence of viruses. Iperium 
> accepts no liability for any damage caused by any virus transmitted by this 
> email.

> From: AusNOG>>On Behalf Of Matt Perkins

> Sent: Tuesday, 14 March 2023 11:18 AM

> To: ausnog@lists.ausnog.net

> Subject: Re: [AusNOG] Vocus Datacentre outage.



> 

> Yea because Eqinix has never had a power outage. :) 

> I think it's a little premature to judge at the moment. Doesn't sound good 
> based on the current info but we have all had bad days im sure. 

> Matt

> 

> On 14/3/2023 11:06 am, Bradley Amm wrote:


>> Friends like let friends put gear in a Vocus dtacenter. Only NextDC or 
>> Equinix :) 

>> Sent from my iPhone





>>> On 14 Mar 2023, at 7:08 am, DaZZa>>> wrote:




>>> Doesn't say a whole lot about Vocus' data center design methodology, nor 
>>> their periodic contingency testing.


>>> 


>>> They're apparently replacing the entire UPS on Friday - guess we can all 
>>> expect more outages then when they stuff that up.


>>> 


>>> D



>>> 

>>> On Mon, 13 Mar 2023 at 18:30, Chad Kelly  wrote:


 Thought I would create a new thread on this, as the issue wasn’t related 
 to AussieBB.

 It is looking like this Melbourne outage