Being slightly serious for a moment, it's a pretty good illustration of how something like a SAN in isolation is no use :-)
-----Original Message----- From: John Aldrich [mailto:jaldr...@blueridgecarpet.com] Sent: 08 October 2010 13:43 To: NT System Admin Issues Subject: RE: How'd this for a bad day? AKA bad me All I can say is "OUCH!" :-( From: David Lum [mailto:david....@nwea.org] Sent: Friday, October 08, 2010 5:51 AM To: NT System Admin Issues Subject: How'd this for a bad day? AKA bad me I have 7 production systems running on 3 different ESX boxes in an ESX cluster, and 2 different logical SAN volumes (sorry am not SAN savvy, I just know I have two different SAN volumes to choose from when making a VM). Today, a SAN blows up and takes out half - our SharePoint server (heavily used), a Terminal Server , and an internal occasionally-used web server (Namescape rDirectory). Then somehow, when I was told to power down the other 4 VM's so our VMWare guy could reboot a vCenter server, 3 of the 4 remaining VM's decided to go AWOL (a combination of "missing" and "disconnected"). That took out my other two Terminal Servers and another lightly used internal web server. Did I mention I don't have the normal backups for these things because ...well...I'm an idiot and didn't confirm our backup guy installed backup software on these servers as I stood them up (process error on my part since I should confirm it's on there). None of these store data - they all talk to a backend SQL and the Terminal Servers are used to run apps that are slow if they run the same apps over VPN. SharePoint we got back quick because we do have a staging equivalent of it, so it was repoint to a config and content DB, DNS change, and done. I do have copious notes on how I built the others and can rebuild from scratch easily enough (I just finished the three TS boxes), but dude...six servers at once? The most frustrating part was discovering that the 4 systems that had been powered off could have been "migrated" before power off and there would have been no issue with them - the power down nuked 'em. Oh, and the lone surviving server - the PGP Universal Server that manages the encrypted machines. (Yes, the PGP machines will still boot w/out the server up, but still, I've been on this server 50% of my time over the last two weeks!). Dave ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com with the body: unsubscribe ntsysadmin ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com with the body: unsubscribe ntsysadmin -- MIRA Ltd Watling Street, Nuneaton, Warwickshire, CV10 0TU, England Registered in England and Wales No. 402570 VAT Registration GB 114 5409 96 The contents of this e-mail are confidential and are solely for the use of the intended recipient. If you receive this e-mail in error, please delete it and notify us either by e-mail, telephone or fax. You should not copy, forward or otherwise disclose the content of the e-mail as this is prohibited. ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com with the body: unsubscribe ntsysadmin
