Amazingly everything is running. So far only one hard drive out of an array is bad. But I am sure we will be really paying for this for months/years to come as this stuff dies early and in weird ways.
Yea, funds are wide open now. I have been on them for two years about this system. It is a brand new building and the HVAC/Emergency power has never been right. This is the third or fourth time this has happened. But in the past they were minor events like you describe. We were hear and were able to minimize the damage. I think it went better this time because my boss is on vacation. He is a great guy but he won't get out the clue by four as often as he should. Nor will he hit them with it as hard as he should. Since he is gone my immediate supervisor is the man that runs the district...of course I had to contact him directly to let him know we were going down hard. I explained the history in a very firm but professional manner. He has already directed operations to do it my way. From: Sean Martin [mailto:seanmarti...@gmail.com] Sent: Thursday, October 18, 2012 4:43 PM To: NT System Admin Issues Subject: Re: Otish Server room disaster. Wow, I've dealt with heat issues before but always in a scenario when we've been able to react within a couple of hours and avoid massive outages. Sounds like this is one of those unfortunate wake calls that magically releases the funds to build in the appropriate level of monitoring and redundancy. Hopefully you were able to recover most of the systems? - Sean On Thu, Oct 18, 2012 at 12:26 PM, Kennedy, Jim <kennedy...@elyriaschools.org<mailto:kennedy...@elyriaschools.org>> wrote: So I come in this morning and things are not well. My first thought is my DNS issues are back from yesterday. But I stay calm and decide to walk the server room first. It is the data center for the whole district. 30 servers, 6509 for all 12 building connects, associated security cameras, video system for the whole district. A lot of gear. Its 125.7 F in the room. Fire Alarm false positive last night at 10:30, that shuts down the HVAC. It is not supposed to shut down the data center but it did. Much of it thermal shut down itself. The first system to blow up was the system to page us when things go bad, no pages. My stuff cooked all night long. We were down most of the day. Sounds like we are getting a second AC with it's own generator and a separate environment monitoring system that uses cell phone to call and kills the power to the room if the temp climbs. ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com<mailto:listmana...@lyris.sunbeltsoftware.com> with the body: unsubscribe ntsysadmin ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com<mailto:listmana...@lyris.sunbeltsoftware.com> with the body: unsubscribe ntsysadmin ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com with the body: unsubscribe ntsysadmin
