Re: Otish Server room disaster.

[Andrew S. Baker]

>>But I am sure we will be really paying for this for months/years to come
as this stuff dies early and in weird ways.
Yes.


>>Yea, funds are wide open now.

Yes, but now the number will be worse than before.
* *

*ASB**http://XeeMe.com/AndrewBaker* *Harnessing the Advantages of
Technology for the SMB market…

*



On Thu, Oct 18, 2012 at 4:50 PM, Kennedy, Jim
wrote:

>  Amazingly everything is running. So far only one hard drive out of an
> array is bad. But I am sure we will be really paying for this for
> months/years to come as this stuff dies early and in weird ways.
>
> ** **
>
> Yea, funds are wide open now. I have been on them for two years about this
> system. It is a brand new building and the HVAC/Emergency power has never
> been right. This is the third or fourth time this has happened. But in the
> past they were minor events like you describe. We were hear and were able
> to minimize the damage.
>
> 
>
> I think it went better this time because my boss is on vacation. He is a
> great guy but he won’t get out the clue by four as often as he should. Nor
> will he hit them with it as hard as he should. Since he is gone my
> immediate supervisor is the man that runs the district…of course I had to
> contact him directly to let him know we were going down hard.
>
> ** **
>
> I explained the history in a very firm but professional manner. He has
> already directed operations to do it my way.
>
> ** **
>
> *From:* Sean Martin [mailto:seanmarti...@gmail.com]
> *Sent:* Thursday, October 18, 2012 4:43 PM
>
> *To:* NT System Admin Issues
> *Subject:* Re: Otish Server room disaster.
>
> ** **
>
> Wow, I've dealt with heat issues before but always in a scenario when
> we've been able to react within a couple of hours and avoid massive
> outages. Sounds like this is one of those unfortunate wake calls that
> magically releases the funds to build in the appropriate level of
> monitoring and redundancy. Hopefully you were able to recover most of the
> systems?
>
>  
>
> - Sean
>
> On Thu, Oct 18, 2012 at 12:26 PM, Kennedy, Jim <
> kennedy...@elyriaschools.org> wrote:
>
> So I come in this morning and things are not well. My first thought is my
> DNS issues are back from yesterday. But I stay calm and decide to walk the
> server room first. It is the data center for the whole district.  30
> servers, 6509 for all 12 building connects, associated security cameras,
> video system for the whole district. A lot of gear.
>
> Its 125.7 F in the room.
>
> Fire Alarm false positive last night at 10:30, that shuts down the HVAC.
> It is not supposed to shut down the data center but it did. Much of it
> thermal shut down itself. The first system to blow up was the system to
> page us when things go bad, no pages. My stuff cooked all night long. We
> were down most of the day.
>
> Sounds like we are getting a second AC with it’s own generator and a
> separate environment monitoring system that uses cell phone to call and
> kills the power to the room if the temp climbs.
>

~ Finally, powerful endpoint security that ISN'T a resource hog! ~
~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/>  ~

---
To manage subscriptions click here: 
http://lyris.sunbelt-software.com/read/my_forums/
or send an email to listmana...@lyris.sunbeltsoftware.com
with the body: unsubscribe ntsysadmin

RE: Otish Server room disaster.

[Ziots, Edward]

It always takes a disaster for folks to open their eyes. Why it has to
be that way... who knows. ( I do but not going to kick that dead horse
anymore than it needs to be)

 

Z

 

Edward E. Ziots, CISSP, Security +, Network +

Security Engineer

Lifespan Organization

ezi...@lifespan.org

 

From: Kennedy, Jim [mailto:kennedy...@elyriaschools.org] 
Sent: Thursday, October 18, 2012 4:51 PM
To: NT System Admin Issues
Subject: RE: Otish Server room disaster.

 

Amazingly everything is running. So far only one hard drive out of an
array is bad. But I am sure we will be really paying for this for
months/years to come as this stuff dies early and in weird ways.

 

Yea, funds are wide open now. I have been on them for two years about
this system. It is a brand new building and the HVAC/Emergency power has
never been right. This is the third or fourth time this has happened.
But in the past they were minor events like you describe. We were hear
and were able to minimize the damage.

 

I think it went better this time because my boss is on vacation. He is a
great guy but he won't get out the clue by four as often as he should.
Nor will he hit them with it as hard as he should. Since he is gone my
immediate supervisor is the man that runs the district...of course I had
to contact him directly to let him know we were going down hard.

 

I explained the history in a very firm but professional manner. He has
already directed operations to do it my way.

 

From: Sean Martin [mailto:seanmarti...@gmail.com] 
Sent: Thursday, October 18, 2012 4:43 PM
To: NT System Admin Issues
Subject: Re: Otish Server room disaster.

 

Wow, I've dealt with heat issues before but always in a scenario when
we've been able to react within a couple of hours and avoid massive
outages. Sounds like this is one of those unfortunate wake calls that
magically releases the funds to build in the appropriate level of
monitoring and redundancy. Hopefully you were able to recover most of
the systems?

 

- Sean

On Thu, Oct 18, 2012 at 12:26 PM, Kennedy, Jim
 wrote:

So I come in this morning and things are not well. My first thought is
my DNS issues are back from yesterday. But I stay calm and decide to
walk the server room first. It is the data center for the whole
district.  30 servers, 6509 for all 12 building connects, associated
security cameras, video system for the whole district. A lot of gear.

Its 125.7 F in the room.

Fire Alarm false positive last night at 10:30, that shuts down the HVAC.
It is not supposed to shut down the data center but it did. Much of it
thermal shut down itself. The first system to blow up was the system to
page us when things go bad, no pages. My stuff cooked all night long. We
were down most of the day.

Sounds like we are getting a second AC with it's own generator and a
separate environment monitoring system that uses cell phone to call and
kills the power to the room if the temp climbs.

~ Finally, powerful endpoint security that ISN'T a resource hog! ~
~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/>  ~

---
To manage subscriptions click here:
http://lyris.sunbelt-software.com/read/my_forums/
or send an email to listmana...@lyris.sunbeltsoftware.com
with the body: unsubscribe ntsysadmin

 

~ Finally, powerful endpoint security that ISN'T a resource hog! ~
~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/>  ~

---
To manage subscriptions click here:
http://lyris.sunbelt-software.com/read/my_forums/
or send an email to listmana...@lyris.sunbeltsoftware.com
with the body: unsubscribe ntsysadmin

~ Finally, powerful endpoint security that ISN'T a resource hog! ~
~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/>  ~

---
To manage subscriptions click here:
http://lyris.sunbelt-software.com/read/my_forums/
or send an email to listmana...@lyris.sunbeltsoftware.com
with the body: unsubscribe ntsysadmin


~ Finally, powerful endpoint security that ISN'T a resource hog! ~
~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/>  ~

---
To manage subscriptions click here: 
http://lyris.sunbelt-software.com/read/my_forums/
or send an email to listmana...@lyris.sunbeltsoftware.com
with the body: unsubscribe ntsysadmin

RE: Otish Server room disaster.

[Kennedy, Jim]

Amazingly everything is running. So far only one hard drive out of an array is 
bad. But I am sure we will be really paying for this for months/years to come 
as this stuff dies early and in weird ways.

Yea, funds are wide open now. I have been on them for two years about this 
system. It is a brand new building and the HVAC/Emergency power has never been 
right. This is the third or fourth time this has happened. But in the past they 
were minor events like you describe. We were hear and were able to minimize the 
damage.
I think it went better this time because my boss is on vacation. He is a great 
guy but he won't get out the clue by four as often as he should. Nor will he 
hit them with it as hard as he should. Since he is gone my immediate supervisor 
is the man that runs the district...of course I had to contact him directly to 
let him know we were going down hard.

I explained the history in a very firm but professional manner. He has already 
directed operations to do it my way.

From: Sean Martin [mailto:seanmarti...@gmail.com]
Sent: Thursday, October 18, 2012 4:43 PM
To: NT System Admin Issues
Subject: Re: Otish Server room disaster.

Wow, I've dealt with heat issues before but always in a scenario when we've 
been able to react within a couple of hours and avoid massive outages. Sounds 
like this is one of those unfortunate wake calls that magically releases the 
funds to build in the appropriate level of monitoring and redundancy. Hopefully 
you were able to recover most of the systems?

- Sean
On Thu, Oct 18, 2012 at 12:26 PM, Kennedy, Jim 
mailto:kennedy...@elyriaschools.org>> wrote:

So I come in this morning and things are not well. My first thought is my DNS 
issues are back from yesterday. But I stay calm and decide to walk the server 
room first. It is the data center for the whole district.  30 servers, 6509 for 
all 12 building connects, associated security cameras, video system for the 
whole district. A lot of gear.

Its 125.7 F in the room.

Fire Alarm false positive last night at 10:30, that shuts down the HVAC. It is 
not supposed to shut down the data center but it did. Much of it thermal shut 
down itself. The first system to blow up was the system to page us when things 
go bad, no pages. My stuff cooked all night long. We were down most of the day.

Sounds like we are getting a second AC with it's own generator and a separate 
environment monitoring system that uses cell phone to call and kills the power 
to the room if the temp climbs.

~ Finally, powerful endpoint security that ISN'T a resource hog! ~
~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/>  ~

---
To manage subscriptions click here: 
http://lyris.sunbelt-software.com/read/my_forums/
or send an email to 
listmana...@lyris.sunbeltsoftware.com<mailto:listmana...@lyris.sunbeltsoftware.com>
with the body: unsubscribe ntsysadmin


~ Finally, powerful endpoint security that ISN'T a resource hog! ~
~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/>  ~

---
To manage subscriptions click here: 
http://lyris.sunbelt-software.com/read/my_forums/
or send an email to 
listmana...@lyris.sunbeltsoftware.com<mailto:listmana...@lyris.sunbeltsoftware.com>
with the body: unsubscribe ntsysadmin

~ Finally, powerful endpoint security that ISN'T a resource hog! ~
~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/>  ~

---
To manage subscriptions click here: 
http://lyris.sunbelt-software.com/read/my_forums/
or send an email to listmana...@lyris.sunbeltsoftware.com
with the body: unsubscribe ntsysadmin

Re: Otish Server room disaster.

[Richard Stovall]

Ouch.  Sorry to hear about the troubles.

On Thu, Oct 18, 2012 at 4:26 PM, Kennedy, Jim
wrote:

>  So I come in this morning and things are not well. My first thought is
> my DNS issues are back from yesterday. But I stay calm and decide to walk
> the server room first. It is the data center for the whole district.  30
> servers, 6509 for all 12 building connects, associated security cameras,
> video system for the whole district. A lot of gear.
>
> Its 125.7 F in the room.
>
> Fire Alarm false positive last night at 10:30, that shuts down the HVAC.
> It is not supposed to shut down the data center but it did. Much of it
> thermal shut down itself. The first system to blow up was the system to
> page us when things go bad, no pages. My stuff cooked all night long. We
> were down most of the day.
>
> Sounds like we are getting a second AC with it’s own generator and a
> separate environment monitoring system that uses cell phone to call and
> kills the power to the room if the temp climbs.
>
> ~ Finally, powerful endpoint security that ISN'T a resource hog! ~
> ~   ~
>
> ---
> To manage subscriptions click here:
> http://lyris.sunbelt-software.com/read/my_forums/
> or send an email to listmana...@lyris.sunbeltsoftware.com
> with the body: unsubscribe ntsysadmin
>

~ Finally, powerful endpoint security that ISN'T a resource hog! ~
~   ~

---
To manage subscriptions click here: 
http://lyris.sunbelt-software.com/read/my_forums/
or send an email to listmana...@lyris.sunbeltsoftware.com
with the body: unsubscribe ntsysadmin

Re: Otish Server room disaster.

[Sean Martin]

Wow, I've dealt with heat issues before but always in a scenario when we've
been able to react within a couple of hours and avoid massive outages.
Sounds like this is one of those unfortunate wake calls that magically
releases the funds to build in the appropriate level of monitoring and
redundancy. Hopefully you were able to recover most of the systems?

- Sean

On Thu, Oct 18, 2012 at 12:26 PM, Kennedy, Jim  wrote:

>  So I come in this morning and things are not well. My first thought is
> my DNS issues are back from yesterday. But I stay calm and decide to walk
> the server room first. It is the data center for the whole district.  30
> servers, 6509 for all 12 building connects, associated security cameras,
> video system for the whole district. A lot of gear.
>
> Its 125.7 F in the room.
>
> Fire Alarm false positive last night at 10:30, that shuts down the HVAC.
> It is not supposed to shut down the data center but it did. Much of it
> thermal shut down itself. The first system to blow up was the system to
> page us when things go bad, no pages. My stuff cooked all night long. We
> were down most of the day.
>
> Sounds like we are getting a second AC with it’s own generator and a
> separate environment monitoring system that uses cell phone to call and
> kills the power to the room if the temp climbs.
>
> ~ Finally, powerful endpoint security that ISN'T a resource hog! ~
> ~   ~
>
> ---
> To manage subscriptions click here:
> http://lyris.sunbelt-software.com/read/my_forums/
> or send an email to listmana...@lyris.sunbeltsoftware.com
> with the body: unsubscribe ntsysadmin
>

~ Finally, powerful endpoint security that ISN'T a resource hog! ~
~   ~

---
To manage subscriptions click here: 
http://lyris.sunbelt-software.com/read/my_forums/
or send an email to listmana...@lyris.sunbeltsoftware.com
with the body: unsubscribe ntsysadmin

RE: Otish Server room disaster.

[Chinnery, Paul]

You have my sympathy, Jim. We had a similar issue but luckily the servers just 
shut down when their internal temp >100.  We did almost the exact same thing 
you're planning.  We put in a new A\C unit and set up a heat\humidty alarm 
system.  It sends an alert to the Operator who then pages us and Plant 
Operations.

From: Kennedy, Jim [mailto:kennedy...@elyriaschools.org]
Sent: Thursday, October 18, 2012 4:27 PM
To: NT System Admin Issues
Subject: Otish Server room disaster.


So I come in this morning and things are not well. My first thought is my DNS 
issues are back from yesterday. But I stay calm and decide to walk the server 
room first. It is the data center for the whole district.  30 servers, 6509 for 
all 12 building connects, associated security cameras, video system for the 
whole district. A lot of gear.

Its 125.7 F in the room.

Fire Alarm false positive last night at 10:30, that shuts down the HVAC. It is 
not supposed to shut down the data center but it did. Much of it thermal shut 
down itself. The first system to blow up was the system to page us when things 
go bad, no pages. My stuff cooked all night long. We were down most of the day.

Sounds like we are getting a second AC with it's own generator and a separate 
environment monitoring system that uses cell phone to call and kills the power 
to the room if the temp climbs.

~ Finally, powerful endpoint security that ISN'T a resource hog! ~
~   ~

---
To manage subscriptions click here: 
http://lyris.sunbelt-software.com/read/my_forums/
or send an email to 
listmana...@lyris.sunbeltsoftware.com
with the body: unsubscribe ntsysadmin

~ Finally, powerful endpoint security that ISN'T a resource hog! ~
~   ~

---
To manage subscriptions click here: 
http://lyris.sunbelt-software.com/read/my_forums/
or send an email to listmana...@lyris.sunbeltsoftware.com
with the body: unsubscribe ntsysadmin