Re: POE switches and lightning
On Thursday 13 May 2010 11:36:35 am Caleb Tennis wrote: I was just curious if anyone had seen anything similar to this before? Our incoming electrical power has surge suppression, and the power to the switches is all through double conversion UPS, so I'm not quite sure why any of them would have been impacted at all. I'm guessing that the strike had some impact on the electrical ground, but I don't know what we can do to prevent future strikes from causing the same issues. Thoughts? Inductively coupled EMP onto the CAT5. I've seen ethernet port chips vaporized on switches. I've even seen holes blown in port interface chips, and the switch continue working (have a DC powered Catalyst 2900XL switch with the center 8 ports in a nonworking state due to EMP from a close strike; the 2900XL is still running fine, just can't use those center eight ports anymore). The building it is installed in is on solar power, and at the time was off- grid. A Siteplayer Telnet was blown, and the eight ports were fried (one of which was connected to the Siteplayer Telnet that got blown) on the switch, but that was the extent of the damage. I'm from a broadcast engineering background, and have seen lightning's effects in many many devices, including vaporized PC traces, etc. Virtually all damage I've seen has been due to either EMP or improperly bonded grounding systems. In particular, if your telecom ground isn't bonded to the electrical NEC safety ground, you will get a voltage difference between the grounds, depending upon the voltage gradient in the ground. Whole books have been written on this subject; I've got one by Polyphaser about nuclear EMP (same concept, larger scale) protection for radio stations. Imagine the lightning bolt's ionization conduction channel as the primary side of many transformers, with every single conductor within many meters being potential secondaries. The closer the secondary, the more coupling. It's a 1:1 turns ratio, too, and so a 100% coupled secondary would give an equal amperage through the secondary. Air-core transformers are loosely coupled at best, but even a tenth of one percent coupling of a 100kiloampere lightning stroke is 100 amps in magnitude. Loosely coupled current transformers, like this, tend to generate large open circuit voltages, too. The most graphic evidence I've seen of the power of lightning-created EMP was made during a strike I saw in June of 1998 at a radio station's studios. The studios were in an old, 1950's vintage school building, built to 1950's civil defense standards for EMP resistance (rebar in a Faraday cage arrangement, metal roof, lightning rods on the roof). There is a 100 foot studio- transmitter link (STL) tower at one end of the building. The STL tower took a direct hit. The Faraday cage rebar verticals embedded in the walls became coupled secondaries, and large currents flowed. Every single CRT monitor in the entire 300 foot long building was left with a rainbow effect on the screen due to the residual magnetism from the EMP. Even monitors that weren't plugged in were rainbowed. Many PC's died that day, but I resurrected several hard drives where I could find identical control boards; no hard drive was unreadable due to magnetic issues, but only electrical (no bad heads or erased sections on the platters; every one I found a compatible replacement control board for was recovered). Made some good money degaussing CRT's that week. (used a bulk tape eraser; turned on the eraser, brought it close to the CRT, worked it over all surfaces, then slowing drew the eraser away from the CRT, and turned it off). The EMP was strong enough that there were a couple of pieces of spare equipment, located in a room less than 30 feet from the tower, that had lightning damage even though they weren't plugged in or connected to anything. One 250MCM ground wire from the tower was vaporized; there were three, and the other two survived, but with noticeable heat-induced discoloration (they were replaced, and the glassed-up ground rods were as well). Engineering estimates of the stroke current were that it was somewhat greater than 200kiloamperes. One of the STL transmitters was damaged, but on the audio side. Neither of the two STL transmitters sustained any RF output damage thanks to the sacrifice of the two daisy-chained Polyphaser arrestors (the arrestors acted as fuses, and had to be replaced, but they're a lot cheaper than a 950MHz Marti STL-10!). One of the two four foot Marti STL dishes had a melted feed, but the other one, which was lower on the tower (about 85 feet up) was undamaged. Fortunately, neither of the two half-inch heliax runs from the dishes were damaged. The 10base-2 LAN took extensive damage, but not every NIC. The most interesting damage was to the RG-58 cable itself, which had holes blown in it every 30 feet or so. Made a good argument to upgrade to 10Base-T
Re: POE switches and lightning
On May 13, 2010, at 2:26 PM, Mark Mayfield wrote: About a month ago, we had a lightning strike near our main campus. We lost one POE Cisco 3560 completely (apparently blown power supply), and in a separate but nearby building, another 3560 lost the ability to deliver POE, but continued to operate as a switch. Both had to be replaced. Both were on wiring closet type UPS'es with surge suppression, and those were unaffected. Mark -Original Message- From: Caleb Tennis [mailto:caleb.ten...@gmail.com] Sent: Thursday, May 13, 2010 10:37 AM To: North American Network Operators Group Subject: POE switches and lightning We had a lightning strike nearby yesterday that looks to have come inside our facility via a feeder circuit that goes outdoors underground to our facility's gate. What's interesting is that various POE switches throughout the entire building seemed to be affected in that some of their ports they just shut down/off. Rebooting these switches brought everything back to life. It didn't impact anything non-POE, and even then, only impacted some devices. But it was spread across the whole building, across multiple switches. I was just curious if anyone had seen anything similar to this before? Our incoming electrical power has surge suppression, and the power to the switches is all through double conversion UPS, so I'm not quite sure why any of them would have been impacted at all. I'm guessing that the strike had some impact on the electrical ground, but I don't know what we can do to prevent future strikes from causing the same issues. Thoughts? It is not clear to me from the above if there are copper circuits coming into the building, but lightning can certainly zap those as well. In very high impact areas (such as mountaintops or Miami) it is a good idea to mandate that all incoming / outgoing circuits are on fiber, without exception. Marshall Confidentiality Statement: The documents accompanying this transmission contain confidential information that is legally privileged. This information is intended only for the use of the individuals or entities listed above. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or action taken in reliance on the contents of these documents is strictly prohibited. If you have received this information in error, please notify the sender immediately and arrange for the return or destruction of these documents.
Re: POE switches and lightning
We had a lightning strike nearby yesterday that looks to have come inside our facility via a feeder circuit that goes outdoors underground to our facility's gate. Perhaps there was a move of the earth-level relative to the neutral line. I have no idea how neutral-line to earth potential is handled in us, but here in austria we use a so called nullung. That means that the earth-ground potential line of the building (which includes also the lightning conductor) is connected to the neutral power line where it enters the building, keeping this potential-difference low. Theres also a potential between earth ground and the neutral-phase of the online-ups. The ethernet-cables; utp or stp? pannels correctly earthed? Perhaps a electrician should check the earthing. Also all copper lines that enter the building should be protected by lightning protectors. Kind regards, Ingo Flaschberger
Re: POE switches and lightning
On 5/14/2010 16:42, Ingo Flaschberger wrote: We had a lightning strike nearby yesterday that looks to have come inside our facility via a feeder circuit that goes outdoors underground to our facility's gate. Perhaps there was a move of the earth-level relative to the neutral line. I have no idea how neutral-line to earth potential is handled in us, but here in austria we use a so called nullung. That means that the earth-ground potential line of the building (which includes also the lightning conductor) is connected to the neutral power line where it enters the building, keeping this potential-difference low. In the US neutral and earth ground are supposed to be bonded only once at the service entrance. A separate ground from the neutral conductor is carried to sub-panels where is it not bonded. Additional bonding can cause weirdness and will turn the ground into a current carrying conductor. However, an older building I used to be in (built 1978) only gave me a neutral with bonded subs, so you'll run into all kinds of stuff depending on the age of the building. Working at a university was particularly interesting with of the vast range of building ages. ~Seth
Re: POE switches and lightning
On 5/14/2010 19:00, Seth Mattinen wrote: On 5/14/2010 16:42, Ingo Flaschberger wrote: We had a lightning strike nearby yesterday that looks to have come inside our facility via a feeder circuit that goes outdoors underground to our facility's gate. Perhaps there was a move of the earth-level relative to the neutral line. I have no idea how neutral-line to earth potential is handled in us, but here in austria we use a so called nullung. That means that the earth-ground potential line of the building (which includes also the lightning conductor) is connected to the neutral power line where it enters the building, keeping this potential-difference low. In the US neutral and earth ground are supposed to be bonded only once at the service entrance. A separate ground from the neutral conductor is carried to sub-panels where is it not bonded. Additional bonding can cause weirdness and will turn the ground into a current carrying conductor. However, an older building I used to be in (built 1978) only gave me a neutral with bonded subs, so you'll run into all kinds of stuff depending on the age of the building. Working at a university was particularly interesting with of the vast range of building ages. In my experience, each building has a building ground-point at the service entrance, as outlined. I the problem in a campus on some soils is that building grounds might be several volts apart--except during thunder storms when the voltage difference might be (it appears) thousands of volts, and with a lightning strike to one of them many thousands of volts. That is why I argue for glass only between buildings. I don't care how much PoE saves. -- Somebody should have said: A democracy is two wolves and a lamb voting on what to have for dinner. Freedom under a constitutional republic is a well armed lamb contesting the vote. Requiescas in pace o email Ex turpi causa non oritur actio Eppure si rinfresca ICBM Targeting Information: http://tinyurl.com/4sqczs http://tinyurl.com/7tp8ml
RE: POE switches and lightning
My first guess would be the lightning was close enough/powerful enough, to send out an EM Pulse which got picked up by the copper going to the devices. This EM Pulse may have been interpreted at the switchport as the device relinquishing power? Had you tried just unplugging one of the devices from Ethernet, and plugging it back in to reset the PoE exchange? Ken Matlock Network Analyst Exempla Healthcare (303) 467-4671 matlo...@exempla.org -Original Message- From: Caleb Tennis [mailto:caleb.ten...@gmail.com] Sent: Thursday, May 13, 2010 9:37 AM To: North American Network Operators Group Subject: POE switches and lightning We had a lightning strike nearby yesterday that looks to have come inside our facility via a feeder circuit that goes outdoors underground to our facility's gate. What's interesting is that various POE switches throughout the entire building seemed to be affected in that some of their ports they just shut down/off. Rebooting these switches brought everything back to life. It didn't impact anything non-POE, and even then, only impacted some devices. But it was spread across the whole building, across multiple switches. I was just curious if anyone had seen anything similar to this before? Our incoming electrical power has surge suppression, and the power to the switches is all through double conversion UPS, so I'm not quite sure why any of them would have been impacted at all. I'm guessing that the strike had some impact on the electrical ground, but I don't know what we can do to prevent future strikes from causing the same issues. Thoughts?
Re: POE switches and lightning
On 5/13/2010 10:36, Caleb Tennis wrote: We had a lightning strike nearby yesterday that looks to have come inside our facility via a feeder circuit that goes outdoors underground to our facility's gate. What's interesting is that various POE switches throughout the entire building seemed to be affected in that some of their ports they just shut down/off. Rebooting these switches brought everything back to life. It didn't impact anything non-POE, and even then, only impacted some devices. But it was spread across the whole building, across multiple switches. I was just curious if anyone had seen anything similar to this before? Our incoming electrical power has surge suppression, and the power to the switches is all through double conversion UPS, so I'm not quite sure why any of them would have been impacted at all. I'm guessing that the strike had some impact on the electrical ground, but I don't know what we can do to prevent future strikes from causing the same issues. Thoughts? I don't know how to account for this in a PoE world, but when I last managed a campus network, we had major issues (particularly in an active-thunder-storm environment) of severe difference in ground-potential between buildings. The only way we could survive was to connect buildings (including free-standing kiosks) with their own grounds using glass. Does anybody make a CAT 5 1-to-1 isolation transformer? -- Somebody should have said: A democracy is two wolves and a lamb voting on what to have for dinner. Freedom under a constitutional republic is a well armed lamb contesting the vote. Requiescas in pace o email Ex turpi causa non oritur actio Eppure si rinfresca ICBM Targeting Information: http://tinyurl.com/4sqczs http://tinyurl.com/7tp8ml
Re: POE switches and lightning
On 05/13/2010 12:19 PM, Larry Sheldon wrote: On 5/13/2010 10:36, Caleb Tennis wrote: We had a lightning strike nearby yesterday that looks to have come inside our facility via a feeder circuit that goes outdoors underground to our facility's gate. What's interesting is that various POE switches throughout the entire building seemed to be affected in that some of their ports they just shut down/off. Rebooting these switches brought everything back to life. It didn't impact anything non-POE, and even then, only impacted some devices. But it was spread across the whole building, across multiple switches. I was just curious if anyone had seen anything similar to this before? Our incoming electrical power has surge suppression, and the power to the switches is all through double conversion UPS, so I'm not quite sure why any of them would have been impacted at all. I'm guessing that the strike had some impact on the electrical ground, but I don't know what we can do to prevent future strikes from causing the same issues. Thoughts? I don't know how to account for this in a PoE world, but when I last managed a campus network, we had major issues (particularly in an active-thunder-storm environment) of severe difference in ground-potential between buildings. Cat 5 has isolation transformers in or just behind each jack. However, in most equipment the grounds aren't really isolated, and in the case of POE they (mostly) aren't at all. Lightning likes to do interesting things. It can induce a 20kv per few feet gradient (or more) across the ground mesh of a power substation (like 4/0 wire in a mesh of 4 foot squares or so; normally more complicated than that since it has to clear equipment etc...). It likes to eat power supplies in well-grounded equipment and leave cheaper stuff alone. It can hit an antenna, leave the receiver completely intact, and fry the power supply of the next box over. We tended to lose either fluorescent ballasts or the thermostat transformer in our furnace when I lived in an active ham's house in Alabama, the radios tended to live. (you should have seen his coax entry panel (1/4 inch copper sheet, grounded outside)), and stuff got manually disconnected from both antennas and power when a storm was expected (every afternoon :-). It wouldn't surprise me if the first answer was right and either the ground pulse or EMP reset the safety switches in the POE feeders. -- Pete
Re: POE switches and lightning
While the equipment may well be affected by an EM pulse, if the gear returns to normal after a power cycle, then the equipment vendor didn't do their job fully developing the product. A product should be tested to take such pulses and should recover provided it has not suffered a catastrophic failure (and in fact it should contain sufficient protection to avoid such in most cases). In working on one particular router in the lab some years ago, I was verifying some software functionality and the hardware engineer I was working with reached over my shoulder and used a device that delivered a high voltage spike (simulated lightning) to a 10BaseT network port. After I peeled myself off the ceiling (and he stopped laughing), we set to work figuring out how to get the device to self-reset after such a strike. One component, an Ethernet hub chip, got into a confused state. I was able to detect this in software, so we adjusted the product design so that the software could yank the hub chip's reset line. It's unfortunate that products, both hardware and software, receive minimal quality testing these days. Guess it's not a surprise, since buyers seemed to prefer products that were quick to market, with lots of bugs, rather than reliability and resilience. On May 13, 2010, at 12:39 PM, Pete Carah wrote: On 05/13/2010 12:19 PM, Larry Sheldon wrote: On 5/13/2010 10:36, Caleb Tennis wrote: We had a lightning strike nearby yesterday that looks to have come inside our facility via a feeder circuit that goes outdoors underground to our facility's gate. What's interesting is that various POE switches throughout the entire building seemed to be affected in that some of their ports they just shut down/off. Rebooting these switches brought everything back to life. It didn't impact anything non-POE, and even then, only impacted some devices. But it was spread across the whole building, across multiple switches. I was just curious if anyone had seen anything similar to this before? Our incoming electrical power has surge suppression, and the power to the switches is all through double conversion UPS, so I'm not quite sure why any of them would have been impacted at all. I'm guessing that the strike had some impact on the electrical ground, but I don't know what we can do to prevent future strikes from causing the same issues. Thoughts? I don't know how to account for this in a PoE world, but when I last managed a campus network, we had major issues (particularly in an active-thunder-storm environment) of severe difference in ground-potential between buildings. Cat 5 has isolation transformers in or just behind each jack. However, in most equipment the grounds aren't really isolated, and in the case of POE they (mostly) aren't at all. Lightning likes to do interesting things. It can induce a 20kv per few feet gradient (or more) across the ground mesh of a power substation (like 4/0 wire in a mesh of 4 foot squares or so; normally more complicated than that since it has to clear equipment etc...). It likes to eat power supplies in well-grounded equipment and leave cheaper stuff alone. It can hit an antenna, leave the receiver completely intact, and fry the power supply of the next box over. We tended to lose either fluorescent ballasts or the thermostat transformer in our furnace when I lived in an active ham's house in Alabama, the radios tended to live. (you should have seen his coax entry panel (1/4 inch copper sheet, grounded outside)), and stuff got manually disconnected from both antennas and power when a storm was expected (every afternoon :-). It wouldn't surprise me if the first answer was right and either the ground pulse or EMP reset the safety switches in the POE feeders. -- Pete
RE: POE switches and lightning
About a month ago, we had a lightning strike near our main campus. We lost one POE Cisco 3560 completely (apparently blown power supply), and in a separate but nearby building, another 3560 lost the ability to deliver POE, but continued to operate as a switch. Both had to be replaced. Both were on wiring closet type UPS'es with surge suppression, and those were unaffected. Mark -Original Message- From: Caleb Tennis [mailto:caleb.ten...@gmail.com] Sent: Thursday, May 13, 2010 10:37 AM To: North American Network Operators Group Subject: POE switches and lightning We had a lightning strike nearby yesterday that looks to have come inside our facility via a feeder circuit that goes outdoors underground to our facility's gate. What's interesting is that various POE switches throughout the entire building seemed to be affected in that some of their ports they just shut down/off. Rebooting these switches brought everything back to life. It didn't impact anything non-POE, and even then, only impacted some devices. But it was spread across the whole building, across multiple switches. I was just curious if anyone had seen anything similar to this before? Our incoming electrical power has surge suppression, and the power to the switches is all through double conversion UPS, so I'm not quite sure why any of them would have been impacted at all. I'm guessing that the strike had some impact on the electrical ground, but I don't know what we can do to prevent future strikes from causing the same issues. Thoughts? Confidentiality Statement: The documents accompanying this transmission contain confidential information that is legally privileged. This information is intended only for the use of the individuals or entities listed above. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or action taken in reliance on the contents of these documents is strictly prohibited. If you have received this information in error, please notify the sender immediately and arrange for the return or destruction of these documents.
Re: POE switches and lightning
Caleb Tennis wrote: We had a lightning strike nearby yesterday that looks to have come inside our facility via a feeder circuit that goes outdoors underground to our facility's gate. What's interesting is that various POE switches throughout the entire building seemed to be affected in that some of their ports they just shut down/off. Rebooting these switches brought everything back to life. It didn't impact anything non-POE, and even then, only impacted some devices. But it was spread across the whole building, across multiple switches. I was just curious if anyone had seen anything similar to this before? Our incoming electrical power has surge suppression, and the power to the switches is all through double conversion UPS, so I'm not quite sure why any of them would have been impacted at all. I'm guessing that the strike had some impact on the electrical ground, but I don't know what we can do to prevent future strikes from causing the same issues. Thoughts? I use these on any cable that leaves my building. http://www.amazon.com/APC-PNET1GB-ProtectNet-Standalone-Protector/dp/B000BKUSS8 It seems to play well with PoE (I put mine before the injector), and also works well with T1s and POTS. -Paul
Re: POE switches and lightning
On May 13, 2010, at 2:24 04PM, Daniel Senie wrote: While the equipment may well be affected by an EM pulse, if the gear returns to normal after a power cycle, then the equipment vendor didn't do their job fully developing the product. A product should be tested to take such pulses and should recover provided it has not suffered a catastrophic failure (and in fact it should contain sufficient protection to avoid such in most cases). In working on one particular router in the lab some years ago, I was verifying some software functionality and the hardware engineer I was working with reached over my shoulder and used a device that delivered a high voltage spike (simulated lightning) to a 10BaseT network port. After I peeled myself off the ceiling (and he stopped laughing), we set to work figuring out how to get the device to self-reset after such a strike. One component, an Ethernet hub chip, got into a confused state. I was able to detect this in software, so we adjusted the product design so that the software could yank the hub chip's reset line. It's unfortunate that products, both hardware and software, receive minimal quality testing these days. Guess it's not a surprise, since buyers seemed to prefer products that were quick to market, with lots of bugs, rather than reliability and resilience. It's not just a matter of these days -- lightning is awfully hard to deal with, because of how quirky the real-world behavior can be. I had to deal with this a lot in the 1970s on RS-232 lines -- we could never predict what would get fried. Of course, there was also a ground strikes very near my apartment, where the induced current tripped a circuit breaker, blew out a couple of lightbulbs, and and came in through the cable TV line to fry the cable box, fry the impedance-matching transformer, and fry the RF input stage on the television... --Steve Bellovin, http://www.cs.columbia.edu/~smb
Re: POE switches and lightning
On 05/13/2010 02:52 PM, Steven Bellovin wrote: On May 13, 2010, at 2:24 04PM, Daniel Senie wrote: While the equipment may well be affected by an EM pulse, if the gear returns to normal after a power cycle, then the equipment vendor didn't do their job fully developing the product. A product should be tested to take such pulses and should recover provided it has not suffered a catastrophic failure (and in fact it should contain sufficient protection to avoid such in most cases). In working on one particular router in the lab some years ago, I was verifying some software functionality and the hardware engineer I was working with reached over my shoulder and used a device that delivered a high voltage spike (simulated lightning) to a 10BaseT network port. After I peeled myself off the ceiling (and he stopped laughing), we set to work figuring out how to get the device to self-reset after such a strike. One component, an Ethernet hub chip, got into a confused state. I was able to detect this in software, so we adjusted the product design so that the software could yank the hub chip's reset line. Luck. I've needed that kind of reset a few times... It's unfortunate that products, both hardware and software, receive minimal quality testing these days. Guess it's not a surprise, since buyers seemed to prefer products that were quick to market, with lots of bugs, rather than reliability and resilience. That is certainly true (and not entirely modern; you can read about that problem in old roman literature. When was Zen and the art of motorcycle maintainance written? - 1970's); however it is nearly impossible to protect well against close-by lightning. It's not just a matter of these days -- lightning is awfully hard to deal with, because of how quirky the real-world behavior can be. I had to deal with this a lot in the 1970s on RS-232 lines -- we could never predict what would get fried. Of course, there was also a ground strikes very near my apartment, where the induced current tripped a circuit breaker, blew out a couple of lightbulbs, and and came in through the cable TV line to fry the cable box, fry the impedance-matching transformer, and fry the RF input stage on the television... I can second Steve in spades; I used to work for the power company in Alabama... There you learn a LOT more than you ever wanted to know about lightning. Consider that one hit can destroy the inside of a 10Mw 66kv-12kv distribution transformer (I actually saw the strike involved; it was less than a mile from my apartment at the time, and dropped power to me; the apt was fed from an entirely different company... My power came back in a few minutes; the other load took almost a week (they had a redundant feed; it was a hospital, but they ran in a low-power mode till a BIG crane and big lo-boy truck came with another transformer)); how are you going to protect any computer from *that*... -- Pete --Steve Bellovin, http://www.cs.columbia.edu/~smb
