What he said. Unbelievable! Sean Rector, MCSE
-----Original Message----- From: Kurt Buff [mailto:kurt.b...@gmail.com] Sent: Monday, January 31, 2011 4:40 PM To: NT System Admin Issues Subject: Re: Some thoughts for your DR Plan - now with Pics Wow. Just... Wow. On Mon, Jan 31, 2011 at 13:13, James Hill <james.h...@superamart.com.au> wrote: > Here are some pictures of the flooding, the lengths we went to to save some > critical servers and what a data centre/server room that's been under flood > water looks like after it has dried out. > > http://cid-0aa8d56800097048.photos.live.com/play.aspx/DR%20Pics?ref=1 > > I have to say that when I saw the data centre after the water had resided it > really hit home (strangely when I had walked around in knee deep water in > there it hadn't been as upsetting). > > I'd spent the last 3 years replacing every piece of IT infrastructure > Australia wide. The Data Centre was new and we had just finished the IT > Infrastructure replacement project. I'd been in discussions with carriers in > preparation for both a live DR location and also to move the Data Centre. I > didn't pick where it was put and I wanted to move it for two reasons. One > was due to the poor carrier connectivity available in the area, and 2ndly > because there had been flooding in the area before. Nothing to this extreme > but I still figured it could happen. > > -----Original Message----- > From: Roger Wright [mailto:rhw...@gmail.com] > Sent: Tuesday, 1 February 2011 4:48 AM > To: NT System Admin Issues > Subject: Re: Some thoughts for your DR Plan > > James, > > Do you have any photos of your DC underwater that you could share? > This and your write-up will go a long way toward helping our management > understand the risks. > > Thanks... > > > Roger Wright > ___ > > "The internet is a great way to get on the net." - Bob Dole > > > > > On Sun, Jan 30, 2011 at 9:05 PM, James Hill <james.h...@superamart.com.au> > wrote: >> We now have the majority of things restored and up and running. >> Below are just some initial thoughts and ideas that I wanted to share with >> the list. >> It is in no way any form of DR plan nor is it meant to indicate what >> we did or didn't have. It's simply my experiences from our recent DR >> experience written down for the benefits of others. >> >> >> >> Some or none of this may apply to you. I certainly do not regard >> myself as any form of DR expert nor am I the first to have been >> through a real DR experience. However if I am able to provide any >> info that can assist others than I am more than happy to do so. >> >> >> >> · Don't ever think it can't happen, it can. >> >> · You do need a DR location, a live one if possible. >> Convince management of this! >> >> · Build redundancy into your designs of everything. Thanks >> to this all our stores were able to continue to trade even though the >> data centre was under water. >> >> · If you have something in your environment that isn't in >> your backup schedule, add it now, no matter how small it may be. >> >> · Consider that staff with specific duties in your DR plan >> may not be able to assist as they are tending to their own personal >> issues or physical access is simply not available. >> >> · Services you take for granted may simply be not available. >> There were power outages (some for weeks) and communication network outages. >> Phone systems quickly become overloaded in a Disaster, especially >> mobile/cell networks. >> >> · Make allowance for the following in your DR location(for >> relocation of office staff) >> >> o Furniture for staff >> >> o Computers and comms >> >> o Power, can the circuits handle the extra load you will be adding >> to the site? >> >> o Bandwidth >> >> o Air conditioning/heating >> >> · Have remote visibility of your data centre and its >> surroundings >> >> o A camera or two would have shown us the level of the water and >> we could have saved much more equipment. >> >> · Add sensors to your data centre that shuts off the power if >> water is detected. >> >> · Exchange cached mode and offline files provide quick access >> to much critical information. >> >> · Keep critical infrastructure/server build/networking >> documentation in multiple places. >> >> o I had a recent backup at my personal residence. It was >> invaluable in the early stages of our Recovery. >> >> · Data restores >> >> o Do test restores regularly. Environments change all the time >> and maybe something hasn't been added to the backup list for that server. >> >> o Ensure that you can retrieve critical data quickly. Restores >> take time. >> >> o Tapes - do anything to avoid them, if you have to use them have >> multiple tape drives available so that restores can be conducted more >> quickly. >> >> o Have backup backup servers. Especially with the tape catalogues >> available. We saw cataloguing of tapes take 14 hours plus. >> >> o Have an offsite location authorised as a delivery point with >> your Offsite Tape holder. >> >> · Check your emotions at the door. Remain calm and logical, >> consider others needs. The people that are true leaders(that doesn't >> necessarily mean all Managers) should be running the show. Everyone >> else will be looking to them for guidance. >> >> >> >> · Fire and water make fantastic servants, they are horrible masters. >> >> >> >> James. >> >> >> >> >> >> ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ >> <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ >> >> --- >> To manage subscriptions click here: >> http://lyris.sunbelt-software.com/read/my_forums/ >> or send an email to listmana...@lyris.sunbeltsoftware.com >> with the body: unsubscribe ntsysadmin > > ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ > <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ > > --- > To manage subscriptions click here: > http://lyris.sunbelt-software.com/read/my_forums/ > or send an email to listmana...@lyris.sunbeltsoftware.com > with the body: unsubscribe ntsysadmin > > > ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ > <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ > > --- > To manage subscriptions click here: > http://lyris.sunbelt-software.com/read/my_forums/ > or send an email to listmana...@lyris.sunbeltsoftware.com > with the body: unsubscribe ntsysadmin > > ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com with the body: unsubscribe ntsysadmin Information Technology Manager Virginia Opera Association E-Mail: sean.rec...@vaopera.org Phone: (757) 213-4548 (direct line) {+} Subscriptions and tickets are on sale now! The Valkyrie | Madama Butterfly Visit us online at www.VaOpera.org or call 1-866-OPERA-VA The vision of Virginia Opera is to enrich lives through the powerful integration of music, voice and human drama. This e-mail and any attached files are confidential and intended solely for the intended recipient(s). Unless otherwise specified, persons unnamed as recipients may not read, distribute, copy or alter this e-mail. Any views or opinions expressed in this e-mail belong to the author and may not necessarily represent those of Virginia Opera. Although precautions have been taken to ensure no viruses are present, Virginia Opera cannot accept responsibility for any loss or damage that may arise from the use of this e-mail or attachments. {*} ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com with the body: unsubscribe ntsysadmin
