RE: Why do we use facilities with EPO's?

[Tim Donahue]

On Fri, 2007-07-27 at 13:09 -0400, Barry Shein wrote:

> 
> Also, NOT TO BE TOO LITERAL MINDED, but isn't the point of a UPS that
> it has a lot of power even when it's not getting any externally?
> 
> Doesn't hitting an EPO on a UPS at best only reduce the electrical
> hazard of hitting it with water a little bit?

Not an engineer, and not a firefighter, but I think I can answer this
one.  

Yes, hitting the EPO would leave a lot of power potential stored in the
batteries in the UPS.

The point of the EPO is to isolate those batteries from both the
external (possibly high voltage) feed coming into the UPS and to isolate
them from the inverters, transformers, power points (plugs) and whatever
else is used in the build-out getting the power out of the batteries and
to the equipment. 

This achieves a couple things. 

* It potentially removes one of the requirements of combustion
(heat/energy input).  

* If the fire is not within/around the UPS itself, the firefighters can
avoid the power that is still in the room.

* Removes the power from the wiring below your raised floor so water
will not come in contact with the power present during normal operation.

I'm sure that there are others out there that could give you even more
reasons, but these are the ones that come to mind immediately.

---
Tim Donahue

RE: Why do we use facilities with EPO's?

[Barry Shein]

On July 26, 2007 at 18:59 [EMAIL PROTECTED] (Randy Epstein) wrote:
 > 
 > I guess my point was that it's safer to power off a UPS system as best you
 > can before you shoot water at it.  :)  Most likely you are doing this at
 > somewhat close proximity, with step-down transformers nearby, etc.

If you can stroll into the room and look around etc., sure, why not. I
said that in the previous msg. We agree.

The note I was responding to asserted that it was necessary to hit an
EPO before (direct) firefighting could commence, I wasn't saying it
wouldn't be handy in some circumstances, just "not entirely necessary"
(for firefighting.)

But getting to an EPO could be difficult if the room is closed and
it's looking like it might be somewhere in excess of 450F inside in
which case the usual approach is to smash/open a window or door while
the others stand ready with a fully charged hose. Which is why they'll
usually shut down power from outside the building if needed.

By definition a room on fire is a room out of control*.

An important component of firefighting is working fast as fires don't
usually get better by themselves. Well, actually they almost all do
get better eventually on their own, when there's nothing left to burn,
but that's not often an attractive option since the available fuel
could be what you call your neighborhood.

Also, NOT TO BE TOO LITERAL MINDED, but isn't the point of a UPS that
it has a lot of power even when it's not getting any externally?

Doesn't hitting an EPO on a UPS at best only reduce the electrical
hazard of hitting it with water a little bit?


* Interesting aside: In many venues, I know this is true in Boston,
when a fire official declares a building on fire legal title to that
building is automatically transferred to the fire dept until
firefighting operations are declared ended.

-- 
-Barry Shein

The World  | [EMAIL PROTECTED]   | http://www.TheWorld.com
Purveyors to the Trade | Voice: 800-THE-WRLD| Login: Nationwide
Software Tool & Die| Public Access Internet | SINCE 1989 *oo*

Re: Why do we use facilities with EPO's?

[Daryl Jurbala]

On Jul 26, 2007, at 6:59 PM, Randy Epstein wrote:


I guess my point was that it's safer to power off a UPS system as  
best you
can before you shoot water at it.  :)  Most likely you are doing  
this at

somewhat close proximity, with step-down transformers nearby, etc.


Somewhat true.

An EPO not only shuts down the power feed to the UPS, but the UPS  
as well.

Which is a good thing.


The batteries still make pretty colors when you hit them and start  
bridging things that shouldn't be bridged.  But if it's not on fire,  
it is usually avoided by the fire department.


I'm posting on this as a 17 year volunteer fire department member as  
well as a professional (albeit part-time, with the rest of my time  
spent in network ops) fire marshal for a town in PA.


EPOs are great, and as a fire marshal I like them (preventative) but  
they really don't figure in to the picture when I've got my  
firefighter hat (ok, helmet) on - because we just cut mains to  
everything, and generally know what we're looking at and how to  
handle it.  Any building in any reasonably juristiction that has any  
"real" sized UPS most likely has not only a pre-plan so the FD knows  
what is where, but also at least annual inspections.  Chances are  
good the facility also has to hold a permit for the number/capacity  
of the batteries in the unit (per IFC 105.7.2) and most likely the  
fuel storage for the generators  (IFC 105.6.16).  Even in your  
jurisdiction doesn't use that code, IFC and/or it's ancestors provide  
the model code that most of the US operates on, so chances are high  
there are similar restrictions/procedures/permitting requirements.


Daryl

Re: Why do we use facilities with EPO's?

[Alexander Harrowell]

>I fail to see why one couldn't have TWO buttons of the same type
This is done on quite a few lumps of industrial machinery.

>While one of the priest-theologians meant
>well, we learned what happened when holy water is sprinkled into the high
>voltage supply of a gas chromatograph

That's a literal example of what happens when faith and science collide.

More broadly, quote of note from Royal Marine officer after recent floods in
the UK - they were shoring up the walls of a major power-grid switching
station, with water inside the facility and much more outside. "I remembered
electricity and water don't mix, but it wasn't a good moment to think
that.." With 600,000 customers hanging off it, needs must when the devil
drives.

Re: Why do we use facilities with EPO's?

[Mattias Ahnberg]

Alex Rubenstein wrote:

EPO = SPOF = bad. We all know this.


I fail to see why one couldn't have TWO buttons of the same
type that needed to be pressed after one another to close
it down. It is unlikely that someone would trip and touch
two separated buttons (although put close to one another).

Probably some logic behind why we don't have this. Or not?
--
/ahnberg.

RE: Why do we use facilities with EPO's?

[Randy Epstein]

> FWIW, do you imagine that's terribly large for urban firefighters in
> the big scheme of things, not just computer rooms?
> 
> My memory could be wrong but I remember the John Hancock building, 60
> stories, pulls about 1.5MW...I remember Boston Edison mentioning this
> in discussing a design I was working on of a supercomputer facility,
> that we were asking for more power than the hancock building which was
> ok but it presented..."challenges". Factories can pull a lot of power
> also (that room was never built.)
> 
> Anyhow, once you're beyond a pea-shooter I don't think procedures for
> firefighting vary a whole lot, other than some outliers.
> 
>   -b

I guess my point was that it's safer to power off a UPS system as best you
can before you shoot water at it.  :)  Most likely you are doing this at
somewhat close proximity, with step-down transformers nearby, etc.

An EPO not only shuts down the power feed to the UPS, but the UPS as well.
Which is a good thing.

A properly placed EPO and warning signs, as well as proper training of your
customers and vendors should minimize the risks associated with an EPO.

Look, if someone is hell bent to destroy your facility, EPO or not, they
will succeed.

Randy

RE: Why do we use facilities with EPO's?

[Barry Shein]

On July 26, 2007 at 16:25 [EMAIL PROTECTED] (Randy Epstein) wrote:
 > (snip)
 > 
 > > Put another way: Between a 120KVA UPS and a gang of experienced
 > > firefighters with charged hoses I'd put my money on the firefighters
 > > every time.
 > > 
 > > --
 > > -Barry Shein
 > 
 > You realize the UPS systems we're speaking of are much larger?  Usually 480
 > volt, many kVA.

FWIW, do you imagine that's terribly large for urban firefighters in
the big scheme of things, not just computer rooms?

My memory could be wrong but I remember the John Hancock building, 60
stories, pulls about 1.5MW...I remember Boston Edison mentioning this
in discussing a design I was working on of a supercomputer facility,
that we were asking for more power than the hancock building which was
ok but it presented..."challenges". Factories can pull a lot of power
also (that room was never built.)

Anyhow, once you're beyond a pea-shooter I don't think procedures for
firefighting vary a whole lot, other than some outliers.

-b

Re: Why do we use facilities with EPO's?

[George William Herbert]

Barry wrote:
>I worked three years with the boston fire dept, albeit quite a few
>years ago, and rode into many fires and don't generally remember them
>being much concerned about hitting *anything* with a high-pressure
>stream of water if it's on fire.
>
>Remember all those rules you know about not using water on electrical
>or chemical fires? Doesn't really count if you have charged fire hoses
>and know what you're doing except in some special circumstances (they
>did foam things occasionally, very occasionally, foam costs money!)

Around here (Silli Valley) the firefighters I know are pretty aware
of the risks of electricity.  They say that some of them have been
fried by UPSes.

And hazmat; we have the large containers of WMD-grade-toxic silicon fab
gases being shipped around.


-george william herbert
[EMAIL PROTECTED]

RE: Why do we use facilities with EPO's?

[Randy Epstein]

(snip)

> Put another way: Between a 120KVA UPS and a gang of experienced
> firefighters with charged hoses I'd put my money on the firefighters
> every time.
> 
> --
> -Barry Shein

You realize the UPS systems we're speaking of are much larger?  Usually 480
volt, many kVA.

Randy

Re: Why do we use facilities with EPO's?

[Barry Shein]

On July 25, 2007 at 14:49 [EMAIL PROTECTED] (George William Herbert) wrote:
 > 
 > 
 > >Seems like the EPO should be a logical AND with the fire alarm system - 
 > >it only works AFTER you have an existing fire alarm in the building.
 > 
 > 
 > No, no.  If the fire alarm system fails, the fire responders need
 > to be able to hit the EPO and be sure that it works anyways.
 > It has to be an absolute - firefighters have to know that the
 > thing they hit was the only, and right, thing, and that they
 > aren't going to die because they sprayed water on an energized
 > but on fire electrical system backed by a 120 KVA UPS or some
 > such.

I worked three years with the boston fire dept, albeit quite a few
years ago, and rode into many fires and don't generally remember them
being much concerned about hitting *anything* with a high-pressure
stream of water if it's on fire.

Remember all those rules you know about not using water on electrical
or chemical fires? Doesn't really count if you have charged fire hoses
and know what you're doing except in some special circumstances (they
did foam things occasionally, very occasionally, foam costs money!)

If they needed the power out, perhaps due to a gas hazard, they
generally go for the power out in the street, calling in the power co
if there's time or, well, one of the firefighters usually knows how to
cut a building's power, between them they usually know just about
everything they need to know about stuff like that.

I have no doubt if they saw an EPO and the room on fire they'd hit it
immediately, why not, as you say it can only make things safer (plus
or minus emergency lighting working but they should have their own.)

But unless there was an explosion hazard I don't remember there being
much concern. Water pressure and getting the equipment positioned and
working was a concern (after life and limb of course.)

Put another way: Between a 120KVA UPS and a gang of experienced
firefighters with charged hoses I'd put my money on the firefighters
every time.

-- 
-Barry Shein

The World  | [EMAIL PROTECTED]   | http://www.TheWorld.com
Purveyors to the Trade | Voice: 800-THE-WRLD| Login: Nationwide
Software Tool & Die| Public Access Internet | SINCE 1989 *oo*

Re: Why do we use facilities with EPO's?

[Jason LeBlanc]

I do.  Hurricane Wilma, blew the roof off our building, water pouring in 
pooling under the floor and onto the PDUs and UPS (800amps of 480v).  We 
wanted to save the data on the servers, had to hit the EPO to enter the 
room (anyone have an idea of how far that much power would arc?).  It 
was STILL quite scary since the batteries were still charged, I actually 
flipped the breaker on the UPS.  Not fun to be around that much power 
when there is a lot of water.  Only time I've ever seen an EPO hit in 
person.


Jerry Pasker wrote:


I've always wondered who died or was injured and caused the EPO to 
come in to existence.  There have been lots of "EPO caused downtime" 
stories, but does anyone on the NANOG list even have one single "Thank 
God for the EPO" story?  I'll feel better about the general state of 
the world if I know that the EPO actually has a real valid use that 
has been ACTUALLY PROVEN IN PRACTICE rather than just in someone's mind.



-Jerry   

RE: Why do we use facilities with EPO's?

[Howard C. Berkowitz]

-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of
Warren Kumari
Sent: Thursday, July 26, 2007 12:03 PM
To: [EMAIL PROTECTED]
Cc: Roy; nanog@merit.edu
Subject: Re: Why do we use facilities with EPO's?



On Jul 26, 2007, at 12:16 AM, [EMAIL PROTECTED] wrote:


Sometime I really need to write down all of the funny things that  
have happened over the years... Actually, if anyone has other, random  
funny (?!) stories, pass them along and I'll make a compilation


[Howard C. Berkowitz] 

While working at a distinguished university with a religious affiliation, I
learned, as did one of the priest-biologists, not to refer to a piece of
instrumentation as possessed. While one of the priest-theologians meant
well, we learned what happened when holy water is sprinkled into the high
voltage supply of a gas chromatograph. Beckman Instruments was so amused
they didn't charge for equipment abuse not under maintenance contract.

Re: Why do we use facilities with EPO's?

[Warren Kumari]

On Jul 26, 2007, at 12:16 AM, [EMAIL PROTECTED] wrote:


On Wed, 25 Jul 2007 12:43:17 PDT, Roy said:


Funny story about that and the EPO we have here...
...

Story #1



Story #2


Story #3


Story #4

I'm still working at the place mentioned in a previous post -- I was  
only there for 3 months (actually one day less than 3 month, I know  
this because the recruiter only got his commission when I was there  
for at least three months, if I'd know this I would have stuck it out  
for another few days), but have more "funny" stories from this place  
than any other, anyway, onto the story:


One of the server rooms becomes unusable and needs to be rebuilt[0],  
so everything needs to be migrated out of the existing room and into  
new space -- this includes a large APC Symmetra UPS. We shut down the  
UPS and pull all of the batteries out of both it and the expansion  
shelves so that we can move it with a pallet lift. We move everything  
into the new space and its time to put the UPS back together. I  
quickly decide that lifting large numbers of heavy batteries into the  
shelves is not fun, so I show the random helper dude what to do...  
"You pick up this big, heavy thing and put in into this cubbyhole  
type spot, then you connect this large connector and slide the  
battery back, lather, rinse, repeat...".


I watch him do the first one and he seems to have it figured out... I  
wander off to go hook up some fiber or something and peer down the  
corridor every now and then to make sure he still has this under  
control. Surprisingly enough he is managing ok and hasn't wandered  
off to take a nap or anything. He gets down to the last few batteries  
and seems to be having some issues, but I figure he'll work it out,  
so I carry on with what I am doing... I peer down the corridor again  
and he is sitting on the floor with his back braced against  
something, pushing the battery into place with his feet... "Whoa,  
this can't be good", I think, just as there is a LARGE bang, a big  
flash and much smoke and fire


Turns out that for the last battery he managed to get the cables  
caught between the side if the battery and the side of the (sheet- 
metal) case. When it didn't just slide easily back, he pushed it  
really hard and the edge of the case chomped through the cable  
creating a dead short -- this literally vaporized a crescent of metal  
from the case around 5 inches in radius, flung bits of molten case  
and battery leads all over the place and ignited the cardboard that  
we put on the pallet to soften it...


Much hilarity ensues...

Sometime I really need to write down all of the funny things that  
have happened over the years... Actually, if anyone has other, random  
funny (?!) stories, pass them along and I'll make a compilation


W

[0]: Have you ever noticed that places that use gas fire suppression  
systems either have doors that open outwards and / or big dampers  
(like http://www.c-sgroup.com/product_home.php? 
section=explovent&page=3) ? Ever wonder why? :-)



--
With Feudalism, it's your Count that votes.

Re: Why do we use facilities with EPO's?

[Valdis . Kletnieks]

On Wed, 25 Jul 2007 12:43:17 PDT, Roy said:

> > Funny story about that and the EPO we have here...
> > ...
> Story #1

> Story #2

Story #3

So about 4 -5 years ago, we were in the middle of a major renovation of our
server room.  Moving machines all over the place, trying to clear about
6K contiguous square feet of floor space to drop a top-5 supercomputer in.
Upgrading the power, bringing in another 1.5Mw feed, cooling to get the
resulting BTUs *out*, etc.  And we decide it's time to put in a new 600kw
diesel backup generator to replace the old one that was way too small, for
all the non-supercomputer systems in the room.

So we take a multi-hour outage one Saturday for a full powerdown so we can wire
all the new UPS gear in.  And one of our scarier moments is rebooting the Sun
E10K, because it was a bit long in the tooth, and had 400 disk drives, and
hadn't been powered off in so long we weren't sure if it *would* power up again
without field engineering assistance.  And it *had* to come back up, because
it had all the Oracle databases that had all our business records, HR,
student records, everything.  There's a few tense moments - we lose about a
dozen drives, but fortunately they're all in RAID sets and no more than one
drive per set died.  We also notice that we dodged a bullet - the main boot
drive was supposed to be mirrored, but due to a config error, wasn't.

Tuesday, that boot drive is moved, it's now mirrored on 2 drives.

Friday, some construction guys come in to move the main entrance door into the
room - it has to move about 20 feet to the right so you can go *around* the
supercomputer, rather than walk straight into it.  And as per plan, one of them
starts moving the kind of odd light switch junction box next to the door, to
its new location next to the new door. Unfortunately, as *not* per plan, he
fails to double-check with our Facilities team that it's been disarmed first...

5 seconds later, it's very quiet and foggy in the room, as the Halon has dumped
and the interlock with the EPO has killed the power.

Several hours later, we finally get to start powering up the Sun E10K.

The good news:  We only lost 2 drives out of 400 this time, rather than a dozen.

The bad news:  Guess which 2 failed.


pgpWcJoYkImdr.pgp
Description: PGP signature

Re: Why do we use facilities with EPO's?

[Stephen Wilcox]

On Wed, Jul 25, 2007 at 07:47:48PM -0400, David Lesher wrote:
> I've never designed or looked into a EPO installation; but I'm
> astonished such does not use a Normally-Closed pushbutton in a
> fail-to-off circuit.
> 
> Similarly...
> 
> If you have electric locks on your exit doors; every installation
> I have seen has a couple of such aspects:
> 
> a) You must have an exit override. If an electric strike, an
> interior knob is good. If a [Locknetics-style] mag-lock, you
> need an exit button.  That button SHALL be a NC pushbutton in
> series with the magnet. [In other words... No, you can't have
> the pushbutton connected back to some controller box on the 3rd
> floor where it generates an interupt that will drop the lock
> power...  or it's supposed to...]

Sorry I've seen a few that dont have an exit override.

> b) When the building fire drop is pulled, you SHALL drop the lock
> power to the mag locks.

I've seen at least one that does not do this.

> And while local fire codes vary widely; given those were in the
> rules for a USG SCIF I worked in; I somehow doubt you'll be able
> to get more lenient treatment based on the import of the data
> center's operation.

That depends on a bunch of criteria.. override buttons and failure when power 
goes out create significant security risks. If you are a bank or have very 
secure data then you might consider these to be ways in which an intruder might 
compromise your security.

>From what I've seen tho, when you remove the ability to exit in this way then 
>you also find you have a lot of control procedures imposed to avoid 
>unnecessary risk to employees or visitors.

Steve

Re: Why do we use facilities with EPO's?

[David Lesher]

Speaking on Deep Background, the Press Secretary whispered:
> 
> 
> Many years ago, the safety department for my employer made a big stink
> over the fact that the EPO hadn't been tested in a couple of years.  We
> scheduled an outage window, shut everything down.  The facilities guy
> pressed the magic big RED button and NOTHING!  Tracing the problem back,
> there was a blown fuse in the EPO circuit because a wire had shorted.  A
> real safe design!

I've never designed or looked into a EPO installation; but I'm
astonished such does not use a Normally-Closed pushbutton in a
fail-to-off circuit.

Similarly...

If you have electric locks on your exit doors; every installation
I have seen has a couple of such aspects:

a) You must have an exit override. If an electric strike, an
interior knob is good. If a [Locknetics-style] mag-lock, you
need an exit button.  That button SHALL be a NC pushbutton in
series with the magnet. [In other words... No, you can't have
the pushbutton connected back to some controller box on the 3rd
floor where it generates an interupt that will drop the lock
power...  or it's supposed to...]

b) When the building fire drop is pulled, you SHALL drop the lock
power to the mag locks.


And while local fire codes vary widely; given those were in the
rules for a USG SCIF I worked in; I somehow doubt you'll be able
to get more lenient treatment based on the import of the data
center's operation.

-- 
A host is a host from coast to [EMAIL PROTECTED]
& no one will talk to a host that's close[v].(301) 56-LINUX
Unless the host (that isn't close).pob 1433
is busy, hung or dead20915-1433

Re: Why do we use facilities with EPO's?

[David Lesher]

Speaking on Deep Background, the Press Secretary whispered:
> 

{good examples deleted...}

> There are a very few exceptions, but for our practical
> purposes, people really ought to simply go to multiple
> site redundancy rather than thinking about bending
> major safety assumptions in how we operate individual
> buildings.  You may have a few less outages, but you
> may also kill someone.
> 
> What the Navy does on ships, for critical ship safety
> and combat systems; what the FAA does for their radar
> facilities and air traffic control facilities; what Telcos
> do, these are different operating regimes, and there are
> associated higher risk acceptance with the different equipment
> setups and safety procedures.

Indeed. The Navy uses a 'battleshort' at times. This is the
ultimate penny in the fusebox; a shunt around the breakers
on critical systems... such as main gun turret hydraulics &
firing controls.

They are there because the analysis was it better to have some
things catch fire than to be impotent when the Bismark or the
Graf Spay is shooting at you.

[NASA used a similar battleshort at Houston MSC during the final
seconds of the A-11 descent, and other short critical periods.]

In both cases, HIGHLY trained people with LOTS of resources were
on the job. Plus, at least in the Navy case, casualties were an
accepted risk; even if you kill folks in one turret, if you get
a hit that saves your ship.

Needless to say, your insurance company is not likely take
that view, no matter how many pron feeds SLA's your EPO will
break. Nor will the local prosecutor trying a manslaughter case
when a fire-fighter dies.





-- 
A host is a host from coast to [EMAIL PROTECTED]
& no one will talk to a host that's close[v].(301) 56-LINUX
Unless the host (that isn't close).pob 1433
is busy, hung or dead20915-1433

Re: Why do we use facilities with EPO's?

[chuck goolsbee]

If you don't have water-based fire suppression, have normally unoccupied
spaces, and are continuously manned, it's sometimes possible to pass on
having an EPO.  YMMV by inspector.


That is indeed true, as we were able to have ours disconnected, and 
were able to expand our facility without adding EPOs in the new 
datacenter rooms.


We were told that since we met a specific list of criteria, which 
included things like FM200/Ecaro25 fire suppression, solid (not 
raised) floors, electrical wiring done some particular way, and large 
signage indicating where the safety hazards lie for our friends in 
the Tukwila & Seattle Fire Departments... we got a pass on the Big 
Red Button. I imagine those criteria change from place to place, and 
time to time... perhaps even inspector to inspector.


I sleep better at night. Well... A little bit better.



--chuck

Re: Why do we use facilities with EPO's?

[George William Herbert]

>I've always wondered who died or was injured and caused the EPO to 
>come in to existence.  There have been lots of "EPO caused downtime" 
>stories, but does anyone on the NANOG list even have one single 
>"Thank God for the EPO" story?  I'll feel better about the general 
>state of the world if I know that the EPO actually has a real valid 
>use that has been ACTUALLY PROVEN IN PRACTICE rather than just in 
>someone's mind.
>
>
>-Jerry   has the irrational fear about the jumper on the "EPO terminal strip" 
>inside his UPSes coming undone.

A friend of mine is a volunteer firefighter (and ex-ISP CEO, used
to be on the list).  I'm not sure that he'd voluntarily enter
a burning datacenter to put it out if there wasn't an EPO.
Firefighters won't use water on live electrical fires.

If your response plan to a fire in the datacenter is "the building burns
down and hope nobody's inside it still" then you're set.

I've seen someone electrocuted and frozen on the wire in a non-datacenter
setting; we flipped the building breaker, which was further than
an EPO would have been.  It wasn't through his chest, and was only
110 V, so it probably didn't make a difference that it took a
minute to turn it off instead of 10 seconds.  There were no severe
burns, etc.

I've seen equipment catch on fire in a datacenter.  If I hadn't
been able to cut off the power locally, the EPO was the last
line of defense...

People I know have hit the EPO when sprinklers discharged in the
datacenter.

If you're lucky these won't happen to you.  But that's not why
safety rules are put in place.  Unluck happens.


-george william herbert
[EMAIL PROTECTED]

EPO/NEC (was Re: Why do we use facilities with EPO's?)

[Alex Pilosov]

On Wed, 25 Jul 2007, Leo Bicknell wrote:

> What I found interesting is that a single EPO is not a hard and fast
> rule.  They walked me through a twisty maze of the national electric
> code, the national fire code, and local regulations. Through that
> journey, they left me with a rather interesting tidbit.
>
> The more "urban" an area the more likely it is to have strict fire
> codes.  Typically these codes require a single EPO for the entire
> structure, there's no way to compartmentalize to rooms or subsystems.
> However in more rural areas this is often not so, and they had in fact
> built data centers to code WITHOUT a single building EPO in several
> locations.  That's to say there was no EPO, but that it may only affect
> a single room, or even a single device.
> 
> If they can be avoided, why do we put up with them?  Do we really
> want our colo in downtown San Francisco bad enough to take the risk
> of having a single point of failure?  How can we, as engineers, ask
> questions about how many generators, how much fuel, and yet take
> for granted that there is one button on the wall that makes it all
> turn off?  Is it simply that having colo in the middle of the city
> is so convenient that it overrides the increased cost and the reduced
> redundancy that are necessitated by that location?
This is an interesting question.

National Electric Code (NEC) requires EPO. Sort of. Articles 645 and 685
deal with it.

While NEC is not binding on every jurisdiction, almost every US
jurisdiction bases its code on NEC with additions/subtractions. I don't 
know offhand if the local changes deal with EPO much, however, here's some 
food for thought regarding EPO and NEC.

With regard to "putting up with them" - EPOs are designed to protect life,
not property or uptime. If there's a short causing electrical fire because
breaker did not open, firefighter better be sure he can cut the power
*before* stepping next to it.

Here's how NEC works:

1) If a room is designed to comply with Article 645, it must have EPO, 
*except* if it qualifies under Article 685.

Being under Article 645 gives couple of things that are generally not 
permitted otherwise, as follows:

645.4 D) permits underfloor wiring for power, receptacles and 
crossconnects.

645.4 E) "Power cables;  comunications cables; connecting cables;
interconnecting cables; and associated boxes, connectors plugs and
receptacles that are listed as part of, or for, information technology
equipment shall not be required to be secured in place".

In other words, you can have crossconnects that are laying on the floor
(or under raised floor but not otherwise secured), and that is OK, 
normally they'd need to be secured every X feet.

645.17) (too lazy to retype NEC language) You can have PDUs with "multiple 
panelboards within a single cabinet" - not all that clear what exactly 
does it permit (PDUs with multiple breaker panels essentially).

My understanding is that if you are willing to forego things that 
Article 645 permits, you do not have to install EPO. Frankly, I don't see 
all that much logic in 645 requirements and linking it to EPO (except, 
possibly, to make operation of datacenters not in compliance with 645 to 
be annoying enough that everyone would opt to comply with EPO).

The Article 685 exception from EPO applies if "An orderly shutdown is
required to minimize personnel hazard and equipment damage". It is really
intented for industrial (like chemical plants control) systems where EPO
shutoff can cause damage to life/property. I doubt this applies to 
datacenter.

Above is an armchair engineer's understanding. To be sure, you should 
consult a real engineer who can stamp and seal your plans!

-alex

Re: Why do we use facilities with EPO's?

[George William Herbert]

>If they can be avoided, why do we put up with them?

Major electrical fire, or arcing in the datacenter.

Flood in the datacenter.  

Accidental water sprinkler discharge in the datacenter.

Equipment fire that the FM-200 didn't put out, and you want
not to have the sprinklers go off if you can avoid it.

Equipment fire in a room without FM-200 in the first place.

Equipment fire with sprinkler discharge, and you'd really
rather just dry out the equipment than have to replace it all.

Electrical worker who's electrocuted himself and going
to die if you can't make the power go away.


There are a very few exceptions, but for our practical
purposes, people really ought to simply go to multiple
site redundancy rather than thinking about bending
major safety assumptions in how we operate individual
buildings.  You may have a few less outages, but you
may also kill someone.

What the Navy does on ships, for critical ship safety
and combat systems; what the FAA does for their radar
facilities and air traffic control facilities; what Telcos
do, these are different operating regimes, and there are
associated higher risk acceptance with the different equipment
setups and safety procedures.


-george william herbert
[EMAIL PROTECTED]

Re: Why do we use facilities with EPO's?

[George William Herbert]

>Seems like the EPO should be a logical AND with the fire alarm system - 
>it only works AFTER you have an existing fire alarm in the building.


No, no.  If the fire alarm system fails, the fire responders need
to be able to hit the EPO and be sure that it works anyways.
It has to be an absolute - firefighters have to know that the
thing they hit was the only, and right, thing, and that they
aren't going to die because they sprayed water on an energized
but on fire electrical system backed by a 120 KVA UPS or some
such.

Also, one should not wait for fire alarms to go off to de-energize
a room in the clear presence of an electrical fire or major short.
Preventing the fire is better than putting it out.


Telco central offices are somewhat of an exception in many ways,
but just about anyone else should have a real live EPO.


-george william herbert
[EMAIL PROTECTED]

Re: Why do we use facilities with EPO's?

[David Lesher]

A few weeks ago a rural farmer in southern Maryland succumbed
to methane gas in a manure pile. His wife went to rescue him,
and she too collapsed. Their two daughters followed.

All 4 died.

And that is why we have EPO's; to keep the rescuers from become
cascade victims.



-- 
A host is a host from coast to [EMAIL PROTECTED]
& no one will talk to a host that's close[v].(301) 56-LINUX
Unless the host (that isn't close).pob 1433
is busy, hung or dead20915-1433

Re: Why do we use facilities with EPO's?

[Warren Kumari]

On Jul 25, 2007, at 3:35 PM, Patrick W. Gilmore wrote:



On Jul 25, 2007, at 2:03 PM, Tuc at T-B-O-H.NET wrote:


If they can be avoided, why do we put up with them?  Do we really
want our colo in downtown San Francisco bad enough to take the risk
of having a single point of failure?  How can we, as engineers, ask
questions about how many generators, how much fuel, and yet take
for granted that there is one button on the wall that makes it all
turn off?  Is it simply that having colo in the middle of the city
is so convenient that it overrides the increased cost and the  
reduced

redundancy that are necessitated by that location?


You forgot the default "Single Point of Failure" in anything..

HUMANS.


The earth is a SPoF.  Let's put DCs on the moon.

Besides, safety always overrides convenience.  And I don't think  
that is a bad trade off.


Me neither...

Having multiple redundant sites (and a well designed network between  
them) is almost always going to be better than a single, wildly  
redundant site. No matter how much redundancy you build into a single  
site, you cannot (realistically) engineer away things like floods,  
etc. Planning your redundancy and testing it though is very important...


Random anecdote (from a friend, I don't know if it true or not):
Back in the day (before cheap international circuits), a very large  
financial in New York needed connectivity to some branches in Europe,  
so they bought some capacity on a satellite transponder and built  
their own ground-station (not cheap) fairly close to NY. They then  
realized that the needed a redundant ground station in case the first  
one failed or something similar, so the built a second ground- 
station, just outside Jersey City


One of the satellite connectivity failure modes is... rain fade.

W




--
TTFN,
patrick




--
"Does Emacs have the Buddha nature? Why not? It has bloody well  
everything else!"

RE: Why do we use facilities with EPO's?

[Alex Rubenstein]

In fact, an EPO system is a single point of failure...

And, whether or not you need an EPO in your center is wholly up to you,
and how you design your center. 

As mentioned at a recent seminar I went to:

"If you do not need to install non-plenum rated cable below a floor, and
you require boxes under the floor to be secured, and you do not state
NFPA 75 as your standard, then you do not need an EPO as defined by NEC
645."

Only if you want exceptions granted in 645 ("Information Technology
Equipment"), should you have to install an EPO.

EPO = SPOF = bad. We all know this.



> > If they can be avoided, why do we put up with them?  Do we really
> > want our colo in downtown San Francisco bad enough to take the risk
> > of having a single point of failure?  How can we, as engineers, ask
> > questions about how many generators, how much fuel, and yet take
> > for granted that there is one button on the wall that makes it all
> > turn off?  Is it simply that having colo in the middle of the city
> > is so convenient that it overrides the increased cost and the
reduced
> > redundancy that are necessitated by that location?
> >
>   You forgot the default "Single Point of Failure" in anything..
> 
>   HUMANS.
> 
>   Tuc/TBOH

Re: Why do we use facilities with EPO's?

[Jerry Pasker]

I've always wondered who died or was injured and caused the EPO to 
come in to existence.  There have been lots of "EPO caused downtime" 
stories, but does anyone on the NANOG list even have one single 
"Thank God for the EPO" story?  I'll feel better about the general 
state of the world if I know that the EPO actually has a real valid 
use that has been ACTUALLY PROVEN IN PRACTICE rather than just in 
someone's mind.



-Jerry   

Re: Why do we use facilities with EPO's?

[Seth Mattinen]

Leo Bicknell wrote:

I was complaining to some of the power designers during the building
of a major facility that the EPO button represented a single point
of failure, and effectively made all of the redundancy built into
the power system useless.  After all, what's the point of having
two (or more) of anything, if there's one button somewhere that
turns it all off?



If the EPO cuts power to the whole facility, it didn't fail, it worked 
perfectly. The failure is actually Slappy the EPO button-pusher.


~Seth

Re: Why do we use facilities with EPO's?

[Patrick W. Gilmore]

On Jul 25, 2007, at 2:03 PM, Tuc at T-B-O-H.NET wrote:


If they can be avoided, why do we put up with them?  Do we really
want our colo in downtown San Francisco bad enough to take the risk
of having a single point of failure?  How can we, as engineers, ask
questions about how many generators, how much fuel, and yet take
for granted that there is one button on the wall that makes it all
turn off?  Is it simply that having colo in the middle of the city
is so convenient that it overrides the increased cost and the reduced
redundancy that are necessitated by that location?


You forgot the default "Single Point of Failure" in anything..

HUMANS.


The earth is a SPoF.  Let's put DCs on the moon.

Besides, safety always overrides convenience.  And I don't think that  
is a bad trade off.


--
TTFN,
patrick

Re: Why do we use facilities with EPO's?

[Roy]

John C. A. Bambenek wrote:
>
> Funny story about that and the EPO we have here...
> ...
Story #1

Many years ago, the safety department for my employer made a big stink
over the fact that the EPO hadn't been tested in a couple of years.  We
scheduled an outage window, shut everything down.  The facilities guy
pressed the magic big RED button and NOTHING!  Tracing the problem back,
there was a blown fuse in the EPO circuit because a wire had shorted.  A
real safe design!

Story #2

Every few years the EPO buttons would change.  First they were the ones
with the metal ring around the button that protects against accidental
pushing.  Then we would get the mushroom button because it was "safer". 
Invariably someone would trip it and they would change them back.  I
think some guy made some money submitting suggestions to change the
button every few years.

Re: Why do we use facilities with EPO's?

[Mark Radabaugh]

Leo Bicknell wrote:

I was complaining to some of the power designers during the building
of a major facility that the EPO button represented a single point
of failure, and effectively made all of the redundancy built into
the power system useless.  After all, what's the point of having
two (or more) of anything, if there's one button somewhere that
turns it all off?
  
Seems like the EPO should be a logical AND with the fire alarm system - 
it only works AFTER you have an existing fire alarm in the building.


--

Mark Radabaugh
Amplex
419.837.5015 x21
[EMAIL PROTECTED]

Re: Why do we use facilities with EPO's?

[John Curran]

At 12:07 PM -0400 7/25/07, Leo Bicknell wrote:

>The more "urban" an area the more likely it is to have strict fire
>codes.  Typically these codes require a single EPO for the entire
>structure, there's no way to compartmentalize to rooms or subsystems.

For high-availability sites (Tier III, Tier IV per UpTime Institute), EPO's are
one of the most common reasons for outage.  I'd highly recommend APC's
paper 
on the topic.

Short-version is that its a safety issue for room occupants and responders.
More mature codes tend to require such, particularly in the presence of
UPS gear which can be energized unbeknownst to fire fighting personnel.
If you don't have water-based fire suppression, have normally unoccupied
spaces, and are continuously manned, it's sometimes possible to pass on
having an EPO.  YMMV by inspector.

/John

Re: Why do we use facilities with EPO's?

[Sean Donelan]

On Wed, 25 Jul 2007, Leo Bicknell wrote:

What I found interesting is that a single EPO is not a hard and
fast rule.  They walked me through a twisty maze of the national
electric code, the national fire code, and local regulations.
Through that journey, they left me with a rather interesting tidbit.


Well.

An Emergency Power Off button is a NEC (the electrical code adopted in 
most of the USA) trade-off for allowing more flexible wiring practices in 
computer rooms.  If you don't use any of those alternative wiring 
practices, you aren't required to install an EPO (modulo the rare local 
code variation). The problem is some people want to get rid of the EPO, 
but also want to keep using alternative wiring practices.


If you've looked in many computer rooms, you'll see some creative wiring 
practices in use.


You'll notice Telco central offices don't have building EPOs.  Likewise 
Equinix data centers don't have EPOs.  But they have limits on what 
wiring practices can be used in their facilities compared to other

data centers.

Re: Why do we use facilities with EPO's?

[Brandon Galbraith]

On 7/25/07, Leo Bicknell <[EMAIL PROTECTED]> wrote:



The more "urban" an area the more likely it is to have strict fire
codes.





If they can be avoided, why do we put up with them?  Do we really

want our colo in downtown San Francisco bad enough to take the risk
of having a single point of failure?  How can we, as engineers, ask
questions about how many generators, how much fuel, and yet take
for granted that there is one button on the wall that makes it all
turn off?  Is it simply that having colo in the middle of the city
is so convenient that it overrides the increased cost and the reduced
redundancy that are necessitated by that location?



We put up with EPOs for the same reason we put up with water-based fire
suppression systems. Safety. When my firefighter buddy needs to cut through
a wall in a data center, she better damn well be sure she can kill power to
the entire area before she potentially cuts through 400-500V feeds in the
walls. Hence, the EPO.

Now, I know you referred to a single EPO for the entire facility, and how
that's akin to doing brain surgery with a sledgehammer. Perhaps someday
someone will come up with a more surgical method for ensuring power is
removed from the areas it needs to, while other unaffected areas can
continue functioning. Until that point though, I prefer we err on the side
of caution and value someone's life over business continuity.

Almost forget. You mentioned more "urban" areas require the master EPO being
easily accessible in the facility. I don't know what gets more urban than
the San Fran area.

-brandon

Re: Why do we use facilities with EPO's?

[Tuc at T-B-O-H.NET]

> If they can be avoided, why do we put up with them?  Do we really
> want our colo in downtown San Francisco bad enough to take the risk
> of having a single point of failure?  How can we, as engineers, ask
> questions about how many generators, how much fuel, and yet take
> for granted that there is one button on the wall that makes it all
> turn off?  Is it simply that having colo in the middle of the city
> is so convenient that it overrides the increased cost and the reduced
> redundancy that are necessitated by that location?
> 
You forgot the default "Single Point of Failure" in anything..

HUMANS.

Tuc/TBOH

Re: Why do we use facilities with EPO's?

[John C. A. Bambenek]

Funny story about that and the EPO we have here...

We have chilled water cooling in our server rooms.  A couple of years
ago we told the facilities guys there was sand in the lines.  They
didn't believe us.  This went back and forth for a few months until
the lines finally ground to a halt.  They admitted sand was in the
lines.

The bring out an HVAC guy... he closes the valve, opens the pipe,
nothing comes out.  He **opens** the valve, nothing comes out.  He
whacks on the pipes with a wrench, all the sand and lots of water come
out very fast.  By the time I got down there, the ceiling tiles were
drenched and looked more like sponges.  Half the room was soaked.

That would be a good reason to have an EPO right there. :)

j

On 7/25/07, Leo Bicknell <[EMAIL PROTECTED]> wrote:


I was complaining to some of the power designers during the building
of a major facility that the EPO button represented a single point
of failure, and effectively made all of the redundancy built into
the power system useless.  After all, what's the point of having
two (or more) of anything, if there's one button somewhere that
turns it all off?

What I found interesting is that a single EPO is not a hard and
fast rule.  They walked me through a twisty maze of the national
electric code, the national fire code, and local regulations.
Through that journey, they left me with a rather interesting tidbit.

The more "urban" an area the more likely it is to have strict fire
codes.  Typically these codes require a single EPO for the entire
structure, there's no way to compartmentalize to rooms or subsystems.
However in more rural areas this is often not so, and they had in
fact built data centers to code WITHOUT a single building EPO in
several locations.  That's to say there was no EPO, but that it may
only affect a single room, or even a single device.

If they can be avoided, why do we put up with them?  Do we really
want our colo in downtown San Francisco bad enough to take the risk
of having a single point of failure?  How can we, as engineers, ask
questions about how many generators, how much fuel, and yet take
for granted that there is one button on the wall that makes it all
turn off?  Is it simply that having colo in the middle of the city
is so convenient that it overrides the increased cost and the reduced
redundancy that are necessitated by that location?

--
   Leo Bicknell - [EMAIL PROTECTED] - CCIE 3440
PGP keys at http://www.ufp.org/~bicknell/
Read TMBG List - [EMAIL PROTECTED], www.tmbg.org





--
/* [EMAIL PROTECTED] is an alias for all ISC handlers.
   Please include the list in all replies to keep everyone informed.
   You may receive more than one response */

Thanks,
j

Why do we use facilities with EPO's?

[Leo Bicknell]

I was complaining to some of the power designers during the building
of a major facility that the EPO button represented a single point
of failure, and effectively made all of the redundancy built into
the power system useless.  After all, what's the point of having
two (or more) of anything, if there's one button somewhere that
turns it all off?

What I found interesting is that a single EPO is not a hard and
fast rule.  They walked me through a twisty maze of the national
electric code, the national fire code, and local regulations.
Through that journey, they left me with a rather interesting tidbit.

The more "urban" an area the more likely it is to have strict fire
codes.  Typically these codes require a single EPO for the entire
structure, there's no way to compartmentalize to rooms or subsystems.
However in more rural areas this is often not so, and they had in
fact built data centers to code WITHOUT a single building EPO in
several locations.  That's to say there was no EPO, but that it may
only affect a single room, or even a single device.

If they can be avoided, why do we put up with them?  Do we really
want our colo in downtown San Francisco bad enough to take the risk
of having a single point of failure?  How can we, as engineers, ask
questions about how many generators, how much fuel, and yet take
for granted that there is one button on the wall that makes it all
turn off?  Is it simply that having colo in the middle of the city
is so convenient that it overrides the increased cost and the reduced
redundancy that are necessitated by that location?

-- 
   Leo Bicknell - [EMAIL PROTECTED] - CCIE 3440
PGP keys at http://www.ufp.org/~bicknell/
Read TMBG List - [EMAIL PROTECTED], www.tmbg.org


pgpLvtSetOd1U.pgp
Description: PGP signature