James,
Glad to hear things are getting better and back to a sort-of-normal for you. Thanks for taking the time to write down and share your thoughts. I passed your note on to our DR planning team, who appreciated your insights as they say they get great value from hearing "real-world" experiences such as yours. -Malcolm From: James Hill [mailto:james.h...@superamart.com.au] Sent: Sunday, January 30, 2011 20:06 To: NT System Admin Issues Subject: Some thoughts for your DR Plan We now have the majority of things restored and up and running. Below are just some initial thoughts and ideas that I wanted to share with the list. It is in no way any form of DR plan nor is it meant to indicate what we did or didn't have. It's simply my experiences from our recent DR experience written down for the benefits of others. Some or none of this may apply to you. I certainly do not regard myself as any form of DR expert nor am I the first to have been through a real DR experience. However if I am able to provide any info that can assist others than I am more than happy to do so. . Don't ever think it can't happen, it can. . You do need a DR location, a live one if possible. Convince management of this! . Build redundancy into your designs of everything. Thanks to this all our stores were able to continue to trade even though the data centre was under water. . If you have something in your environment that isn't in your backup schedule, add it now, no matter how small it may be. . Consider that staff with specific duties in your DR plan may not be able to assist as they are tending to their own personal issues or physical access is simply not available. . Services you take for granted may simply be not available. There were power outages (some for weeks) and communication network outages. Phone systems quickly become overloaded in a Disaster, especially mobile/cell networks. . Make allowance for the following in your DR location(for relocation of office staff) o Furniture for staff o Computers and comms o Power, can the circuits handle the extra load you will be adding to the site? o Bandwidth o Air conditioning/heating . Have remote visibility of your data centre and its surroundings o A camera or two would have shown us the level of the water and we could have saved much more equipment. . Add sensors to your data centre that shuts off the power if water is detected. . Exchange cached mode and offline files provide quick access to much critical information. . Keep critical infrastructure/server build/networking documentation in multiple places. o I had a recent backup at my personal residence. It was invaluable in the early stages of our Recovery. . Data restores o Do test restores regularly. Environments change all the time and maybe something hasn't been added to the backup list for that server. o Ensure that you can retrieve critical data quickly. Restores take time. o Tapes - do anything to avoid them, if you have to use them have multiple tape drives available so that restores can be conducted more quickly. o Have backup backup servers. Especially with the tape catalogues available. We saw cataloguing of tapes take 14 hours plus. o Have an offsite location authorised as a delivery point with your Offsite Tape holder. . Check your emotions at the door. Remain calm and logical, consider others needs. The people that are true leaders(that doesn't necessarily mean all Managers) should be running the show. Everyone else will be looking to them for guidance. . Fire and water make fantastic servants, they are horrible masters. James. ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com with the body: unsubscribe ntsysadmin ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com with the body: unsubscribe ntsysadmin
