Re: San Francisco Power Outage
On Tue, 24 Jul 2007, Tuc at T-B-O-H.NET wrote: (I remember two guys with VERY LONG screwdrivers poking a live transfer switch to get it to reset properly, and was told to step back 20 feet as thats how far they expected to get thrown if they did something wrong). (I also remember them resetting the switch, then TRIPPING it again just to make sure it could be reset again!) Ahhh, a trip down memory lane :) The ISP I used to work at had a small ping-and-power colo space, and we also housed a large dial/DSL POP in the same building. A customer went in to do hardware maintenance on one of their colo boxes. Two important notes here: 1. The machine was still plugged in to the power outlet when they decided to do this work. 2. They decided to stick a screwdriver into the power supply WHILE said machine was plugged into said power outlet. I guess those no user serviceable parts inside warning labels are just friendly recommendations and nothing more... While the machine was fed from a circuit that other colo customers were on, the breaker apparently didn't trip quickly enough to keep the resulting short from sending the 20 kva Liebert UPS at the back of the room into a fit. It alarmed then shut down within 1-2 seconds of this customer doing the trick with the screwdriver. This UPS also fed said large dial and DSL POP. Nothing quite like the sound of a whole machine room spinning down at the same time. It gives you that lovely oh shit feeling in the pit of your stomach. I do remember fighting back the urge to stab said customer with that screwdriver... jms
RE: more on SF outage
Once the final analysis of this event is provided, it is likely going to be due to a failure of one of the redundant systems to handle the event as designed due to a software or other low level failure. It's a very complex system designed to exceed anything in the region as far as redundancy goes, but as a result it's got a lot of moving parts, and like the space shuttle, can fail unexpectedly. You can bet engineering is scratching their head and calling in the vendors to figure out what went wrong. Last time this occurred it took weeks to pinpoint the root cause.
RE: 365 Main - an operators' nightmare?
I believe this happened to an Internap facility in Seattle a couple of years ago: http://community.livejournal.com/lj_dev/670215.html I was told it happened in our colo facility about a month before we moved in. Some unfortunate remodeling of previous data center space had left an EPO switch in a janitor's closet. The maid knocked loose the protective covering, which of course made an alarm start screaming...so she hit the EPO to stop the noise. Thankfully, the switch has been since removed... Anyhow, any story involving an EPO at 365 Main seems plausible... -J -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jim Popovitch Sent: Tuesday, July 24, 2007 8:59 PM To: Rusty Hodge Cc: [EMAIL PROTECTED] Edu Subject: Re: 365 Main - an operators' nightmare? On Tue, 2007-07-24 at 19:26 -0700, Rusty Hodge wrote: Think that's good? It gets better http://valleywag.com/tech/breaking/angry-mob-gathers-outside-sf- datacenter-282053.php That article states that only Colo 4 was affected. I'm in Colo 7 and it was affected as well. You're not seriously believing the disgruntled employee story are you? No. ;-) But it is otherwise believable. I've seen people hit big-red-buttons in disbelief before, doing so in anger seems very plausible. -Jim p. !SIG:46a6d6e0156535690315935!
RE: An Internet IPv6 Transition Plan
You posit that running out of bread (ipv4 address space) encourages people to bake more bread. Unfortunately it often makes them scream for bread lines (rationing, central control, privilege.) It'd be nice if there were a more positive reason to go ipv6 than getting out of the bread lines, but the killer ipv6 app remains elusive. -- -Barry Shein The World | [EMAIL PROTECTED] | http://www.TheWorld.com Purveyors to the Trade | Voice: 800-THE-WRLD| Login: Nationwide Software Tool Die| Public Access Internet | SINCE 1989 *oo*
Re: ASN Name of the week
On Wed, 25 Jul 2007, Tuc at T-B-O-H.NET wrote: www.1800gotjunk.com. They're all over Canada and the US (at the very least). It's a very successful franchise operation. I don't know why they need an AS, but I can say they did a bang-up job of hauling the detritus out of a condo I used to own after the renter abandoned it. Maybe they'll take away all your unwanted SPAM and DDOS attack traffic. :) Or maybe they are getting large enough that they'll be moving out of their colo centers and into one of their own, multi homed. I just multihomed my house and might apply for an ASN for it... :) (When is ASNV6 coming?) Tuc/TBOH Hi, ASNV6, no clue... but 32-bit ASN are already prepared, at least in the registry world. http://www.arin.net/registration/templates/asn-request.txt - Template: ARIN-ASN-REQUEST-4.0 ** As of July 2006 ** Detailed instructions are located below the template. 01. Org ID: 02. Org Name: 03. AS Name: ** Do you want to specifically request a 4-byte AS number ** instead of a traditional 2-byte AS number? Indicate ** YES or NO. If you are unsure, see detailed instructions ** for an explanation of 2-byte vs. 4-byte AS numbers. 04. 4-byte AS number: (...) Cheers, - Carlos Friac,asSee: Wide Area Network Working Group (WAN) www.gigapix.pt FCCN - Fundacao para a Computacao Cientifica Nacional www.ipv6.eu Av. do Brasil, n.101 www.6diss.org 1700-066 Lisboa, Portugal, Europe www.geant2.net Tel: +351 218440100 Fax: +351 218472167 www.fccn.pt - The end is near see http://ipv4.potaroo.net Internet is just routes (217118/774), naming (billions) and... people! Aviso de Confidencialidade Esta mensagem e' exclusivamente destinada ao seu destinatario, podendo conter informacao CONFIDENCIAL, cuja divulgacao esta' expressamente vedada nos termos da lei. Caso tenha recepcionado indevidamente esta mensagem, solicitamos-lhe que nos comunique esse mesmo facto por esta via ou para o telefone +351 218440100 devendo apagar o seu conteudo de imediato. Warning This message is intended exclusively for its addressee. It may contain CONFIDENTIAL information protected by law. If this message has been received by error, please notify us via e-mail or by telephone +351 218440100 and delete it immediately.
Re: San Francisco Power Outage
From: Justin M. Streiner [EMAIL PROTECTED] Sent: Tuesday, July 24, 2007 5:58 PM Subject: Re: San Francisco Power Outage Nothing quite like the sound of a whole machine room spinning down at the same time. It gives you that lovely oh shit feeling in the pit of your stomach. Yep. I plugged in my soldering iron and (coincidentally) the whole room at State of Calif., Franchise Tax, EPO'd. Everyone immediately started staring at me of course. --Michael
RE: San Francisco Power Outage
And the stories that the power guy I'm working with tells about foreign facilities, particularly in middle east war zones, are really scary... We fundamentally do not have the facilities problem completely nailed down to the point that things will never drop. Level 4 datacenters can, and will, fail. Nothing you can do including just doing 48V DC for everything are truly foolproof solutions. A single level 4 datacenter is a Single Point of Failure! Two of those middle-eastern style facilities is... ? Has anyone actually kept track of all these data center failures over the years and done some statistical analysis on it? Maybe two half-baked data centers is better than one over the long run? Remember that one 10-12 years ago in (Palo Alto, Mountainview?) where a lady in a car caused a backhoe driver to move out of the way which resulted in him cutting a gas line which resulted in the fire department evacuating the data center, cutting off electricity in the area, and forbidding the diesel generators to be switched on? --Michael Dillon
Re: An Internet IPv6 Transition Plan
On Tue, Jul 24, 2007 at 09:34:01PM +0100, [EMAIL PROTECTED] wrote: However, what I'm trying to understand is why the motivation to rapidly go from v4 to v6 only? What are the factors I'm missing in operating v4/v6 combined for some time? Growth. Lack of IPv4 addresses will put the brakes on growth of the Internet which will have a major impact on revenue growth. Before long stock market analysts are going to be asking tough questions, and CEOs are suddenly going to see the IPv6 light. What exactly will cease to grow tho? The 4 billion IPs that have always been around will continue to be. I think you overestimate the effects.. All the existing big businesses can operate with what they already have, Google and Yahoo are not going to face any sort of crisis for the foreseeable future. And as I've been saying for a while and Randy put in his presentation, supply and demand will simply cause the cost of having public IPs to go up from zero to something tiny - enough to see IPs being put back into the pool to those who really need them. Steve
Re: San Francisco Power Outage
On Tue, Jul 24, 2007 at 11:57:37PM +, Paul Vixie wrote: [EMAIL PROTECTED] (Seth Mattinen) writes: I have a question: does anyone seriously accept oh, power trouble as a reason your servers went offline? Where's the generators? UPS? Testing said combination of UPS and generators? What if it was important? I honestly find it hard to believe anyone runs a facility like that and people actually *pay* for it. If you do accept this is a good reason for failure, why? sometimes the problem is in the redundancy gear itself. PAIX lost power twice during its first five years of operation, and both times it was due to faulty GFI in the UPS+redundancy gear. which had passed testing during construction and subsequently, but eventually some component just wore out. I had an issue with exactly that 7 or 8 years ago at Via Networks.. the switchover gear shorted and died horrifically leading to an outage that lasted well through the night (something like 16hours in total). Being on a Friday evening it was difficult to get people on site promptly. The lesson learned was 'the big switch' .. a huge thing that took the weight of two adults to move it, but did mean that should something similar occur we could transfer the whole building power manually directly to the generator. I doubt such a beast would scale to the power loads on a large datacentre tho, but then they are generally not on a single grid/UPS feed. Steve
Re: An Internet IPv6 Transition Plan
At 11:52 AM +0100 7/25/07, Stephen Wilcox wrote: On Tue, Jul 24, 2007 at 09:34:01PM +0100, [EMAIL PROTECTED] wrote: However, what I'm trying to understand is why the motivation to rapidly go from v4 to v6 only? What are the factors I'm missing in operating v4/v6 combined for some time? Growth. Lack of IPv4 addresses will put the brakes on growth of the Internet which will have a major impact on revenue growth. Before long stock market analysts are going to be asking tough questions, and CEOs are suddenly going to see the IPv6 light. What exactly will cease to grow tho? The 4 billion IPs that have always been around will continue to be. I think you overestimate the effects.. All the existing big businesses can operate with what they already have, Google and Yahoo are not going to face any sort of crisis for the foreseeable future. And as I've been saying for a while and Randy put in his presentation, supply and demand will simply cause the cost of having public IPs to go up from zero to something tiny - enough to see IPs being put back into the pool to those who really need them. Steve - Putting them back into circulation doesn't work unless its done in very large chucks to major ISPs. If this isn't the model followed, then we will see a lot more routes for the equivalent number of new customers. People complaining about the ability to carry both IPv6 and IPv4 routing need to think carefully about how long we'll actually last if the ISP's are injecting thousands of unaggregatable routes from recovered address space each day. Additionally, the run rate for IPv4 usage approximates 10 /8 equivalents per year and increasing. Even given great legacy recovery, you've only gained a few more years and then still have to face the problem. /John
RE: An Internet IPv6 Transition Plan
Lack of IPv4 addresses will put the brakes on growth of the Internet which will have a major impact on revenue growth. Before long stock market analysts are going to be asking tough questions, and CEOs are suddenly going to see the IPv6 light. What exactly will cease to grow tho? The 4 billion IPs that have always been around will continue to be. I think you overestimate the effects.. I think you misunderstand the dictionary definition of growth. Yes, the IPv4 addresses, and much of the network infrastructure using them, will continue to be. But growth is about expansion, adding more, increasing the size and scope of the network. Few businesses are satisfied with collecting the same monthly recurring revenue from the same customer base. They either want to grow the customer base or grow the monthly revenue per customer. In the Internet business the main engine of revenue growth is growing the customer base by growing the network and adding more customers. All the existing big businesses can operate with what they already have, Google and Yahoo are not going to face any sort of crisis for the foreseeable future. I disagree. In reality, the customer base of a business is never static. If the company does not grow their base, they certainly will see that base shrink through attrition, churn, etc. Customers will die, move to another town/country, and switch suppliers for some reason or other. In order to keep from fading away, a company has to grow its base, and if there are hard geographic limits to growth because of IPv4 exhaustion, that makes it complex (and therefore expensive) to maintain a steady state. And as I've been saying for a while and Randy put in his presentation, supply and demand will simply cause the cost of having public IPs to go up from zero to something tiny - enough to see IPs being put back into the pool to those who really need them. And when your Internet supplier tells you that there will be a $10 per month increase in fees to cover the increase cost of IPv4 addresses, will you be happy? Will you start shopping for an IPv6 Internet supplier? When IPv6 Internet access is cheaper due to IPv4 address costs, then ISPs face a wholesale loss of their customer base. Of course, most business managers are smart enough to see this coming and resist paying for IPv4 addresses in the first place. Let's face it, the majority of ISP and telecom executives in place today, have spent their careers navigating through a period of growth and abundant resources. They don't know how to manage through scarcity and constraints and shortages. Many of them realize this and will steer their businesses to avoid scarcity and constraints and shortages. That means that most of them will see IPv6 as an opportunity to see who can race the fastest and build market share before the competition does. They know how to do this, and the investment bankers also understand this model of business. When the IPv4 shortage begins to bite, then you will see enormous amounts of money and effort put into IPv6 conversions (and new IPv6 startups who intend to unseat Google, Yahoo, etc.). There's another killer application of IPv6. --Michael Dillon
Re: DNS Hijacking by Cox
Peter Dambier wrote: The problem is, you dont know what is behind that probably NATted ip address. Probably you have 3 unix machines running smtp and uucp and a single infected windows box and maybe some VoIPs and ... This is why I spoke of merely intercepting web traffic to inform, to not interrupt other services that may use the same link. I am in the same situation myself, sharing lots of stuff via the same fiber to my house. I even have TV through it. So I actually thought of that. And an ISP probably knows a bit more about their customer base than what we do, so this idea would ofcourse have to adapt to that. But as said, its a complicated matter and probably not a good idea either way before we know who is supposed to do what and for whom. -- /ahnberg.
Re: An Internet IPv6 Transition Plan
On Wed, Jul 25, 2007 at 07:14:49AM -0400, John Curran wrote: At 11:52 AM +0100 7/25/07, Stephen Wilcox wrote: On Tue, Jul 24, 2007 at 09:34:01PM +0100, [EMAIL PROTECTED] wrote: However, what I'm trying to understand is why the motivation to rapidly go from v4 to v6 only? What are the factors I'm missing in operating v4/v6 combined for some time? Growth. Lack of IPv4 addresses will put the brakes on growth of the Internet which will have a major impact on revenue growth. Before long stock market analysts are going to be asking tough questions, and CEOs are suddenly going to see the IPv6 light. What exactly will cease to grow tho? The 4 billion IPs that have always been around will continue to be. I think you overestimate the effects.. All the existing big businesses can operate with what they already have, Google and Yahoo are not going to face any sort of crisis for the foreseeable future. And as I've been saying for a while and Randy put in his presentation, supply and demand will simply cause the cost of having public IPs to go up from zero to something tiny - enough to see IPs being put back into the pool to those who really need them. Steve - Putting them back into circulation doesn't work unless its done in very large chucks to major ISPs. If this isn't the model followed, then we will see a lot more routes for the equivalent number of new customers. People complaining about the ability to carry both IPv6 and IPv4 routing need to think carefully about how long we'll actually last if the ISP's are injecting thousands of unaggregatable routes from recovered address space each day. Additionally, the run rate for IPv4 usage approximates 10 /8 equivalents per year and increasing. Even given great legacy recovery, you've only gained a few more years and then still have to face the problem. Hi John, I fully agree on that.. but I am disagreeing as to the timescales. There is some opinion that when IANA hands out the last of its IP blocks things will change overnight, and I dont see any reason for that to be the case. I think there are a lot of IPs currently allocated to ISPs but as yet unassigned to customers, and I think there will be a lot of policy changes to make more efficient use of the space that is already out there - I specifically think that will come from ISPs reusing IPs and setting costs that ensure they continually have IPs available to customers willing to pay for them. I think the combined effect of these things means - we will not be running into a wall at any time - availability of IPs will slowly decrease over time (as cost slowly increases) - adoption of NAT and v6 will be an ongoing trend with no sudden increase This means no end of the world as we know it, and no overnight adoption of new technology.. just business as usual in an evolving environment. Steve
Re: Problems getting Cisco router and Motorola Nextlevel system to work together
This router has a G-1 engine with 512 DRAM. I would stop using IRB, but it appears that the way that motorola has implemented pvc's is very difficult to work around. The Molorola middleware is dynamically assigning the pvc. Yes... I have personly seen a CPE device change their vci after a period of time. The device did not change ports or anything else but was provisioned to a different vci after just sitting there. Thanks for the suggestions so far. -- Brian Raaen Network Engineer [EMAIL PROTECTED] On Tuesday 24 July 2007 16:25, you wrote: The router is currently configured to use IRB which is a hybrid process. The problems is that the IRB process is overloaded and is dropping traffic faster than it can process it. Which NPE is in this router? Basically, the 7200 has underpowered CPUs and if you force it to process switch, then it handles a LOT LESS packets per second than you might think. I expect that your config is forcing process switching rather than fast switching. The only three solutions are A) run less traffic through the 7200 so that process switching can cope B) stop using the feature that forces process switching C) replace the 7200 with a 7300 which will probably not have CPU issues. However, not knowing the specifics of what IRB is doing, I would advise you to test a replacement platform before committing to it. Oh well, maybe 4 solutions. If you are using a weak NPE such as NPE-200 you may be able to get some joy by upgrading to a more powerful one. For instance an NPE-400 should handle roughly twice the load of an NPE-200. --Michael Dillon
Re: Problems getting Cisco router and Motorola Nextlevel system to work together
The buffers are overloading and dropping traffic. With a Cisco TAC case, the tech had me increase the buffers so much it wasn't even funny. The only problem was about and hour after we tried to tune the buffers, things got very bad and I had clear them to default to stop a very ugly bigger outage. This system does indeed involve IPTV set top boxes. I am unable to use RBE since the PVC provisioning may change on the units and the VC would not match what the dhcp lease was originally on. The way that this Motorola system implements PVCs baffles me, it does not make any sense to me. They are dynamically changing the vci assigning it out of a pool, just like DHCP does with IPs. The circuits are not SVCs and the endpoint router is seeing things change so this is not SPVCs either. I am trying to think of a way the change this to work with RBE switching, but the dynamic PVCs are throwing a monkey wrench into things. Thank for the help. -- Brian Raaen Network Engineer [EMAIL PROTECTED] On Tuesday 24 July 2007 22:58, you wrote: We should probably move this over to cisco-nsp. I'd be interested to see a 'sh buffers' because if it's process switching that much data I bet the buffers are thrashing. I seem to remember working on something very similar to that 4 or 5 years ago when a customer has brigding over a bunch of ATM PVC's and they told me it was some type of IPTV set top box. We tuned the buffers really high so they didn't trim back and it worked. We also do some bridging under interrupt without process switching too last time I checked so some more data would be helpful. Move it over to [EMAIL PROTECTED] and we can help more on the Cisco side if you want. Rodney On Tue, Jul 24, 2007 at 09:25:49PM +0100, [EMAIL PROTECTED] wrote: The router is currently configured to use IRB which is a hybrid process. The problems is that the IRB process is overloaded and is dropping traffic faster than it can process it. Which NPE is in this router? Basically, the 7200 has underpowered CPUs and if you force it to process switch, then it handles a LOT LESS packets per second than you might think. I expect that your config is forcing process switching rather than fast switching. The only three solutions are A) run less traffic through the 7200 so that process switching can cope B) stop using the feature that forces process switching C) replace the 7200 with a 7300 which will probably not have CPU issues. However, not knowing the specifics of what IRB is doing, I would advise you to test a replacement platform before committing to it. Oh well, maybe 4 solutions. If you are using a weak NPE such as NPE-200 you may be able to get some joy by upgrading to a more powerful one. For instance an NPE-400 should handle roughly twice the load of an NPE-200. --Michael Dillon
Re: An Internet IPv6 Transition Plan
At 12:30 PM +0100 7/25/07, Stephen Wilcox wrote: Hi John, I fully agree on that.. but I am disagreeing as to the timescales. There is some opinion that when IANA hands out the last of its IP blocks things will change overnight, and I dont see any reason for that to be the case. I think there are a lot of IPs currently allocated to ISPs but as yet unassigned to customers, and I think there will be a lot of policy changes to make more efficient use of the space that is already out there - I specifically think that will come from ISPs reusing IPs and setting costs that ensure they continually have IPs available to customers willing to pay for them. In the ARIN region, we've got major ISP's coming back every 6 months with high utilization rates seeking their next block to allow customer growth. While I'm certain that some internal recovery is possible, there's a realistic limit of how long any ISP can make their air supply last. I think the combined effect of these things means - we will not be running into a wall at any time - availability of IPs will slowly decrease over time (as cost slowly increases) - adoption of NAT and v6 will be an ongoing trend with no sudden increase Unless the policy changes you suggest somehow dramatically change the current usage rate, we're going to have a very serious rate of change when the IANA/RIR pool hits zero. That sort of defines hitting a wall, by my definition. Please propose the magical policy changes asap... we need to get them through the public process and adopted in record time to have any affect on the usage rate. This means no end of the world as we know it, and no overnight adoption of new technology.. just business as usual in an evolving environment. Note: I'm not advocating an overnight technology deployment; just advising those folks who presently rely on continuous availability of new address blocks from the RIR's that we're going to see a change. At present, there's a few years for these folks to switch to IPv6 for their growth. It requires cooperation from the Internet, in that we all need to recognize that there will be IPv6 customers out there soon, and even if you don't plan on having those, please make your public facing servers IPv6 reachable in the next few years. /John
Re: An Internet IPv6 Transition Plan
On Wed, Jul 25, 2007, Stephen Wilcox wrote: Lack of IPv4 addresses will put the brakes on growth of the Internet which will have a major impact on revenue growth. Before long stock market analysts are going to be asking tough questions, and CEOs are suddenly going to see the IPv6 light. What exactly will cease to grow tho? The 4 billion IPs that have always been around will continue to be. I think you overestimate the effects.. All the existing big businesses can operate with what they already have, Google and Yahoo are not going to face any sort of crisis for the foreseeable future. And as I've been saying for a while and Randy put in his presentation, supply and demand will simply cause the cost of having public IPs to go up from zero to something tiny - enough to see IPs being put back into the pool to those who really need them. I'm not sure what your definition of really tiny is, but out here IPs are a dollar or two each a year from APNIC. I'm sure ARIN's IP charges aren't $0.00. Adrian
Re: San Francisco Power Outage
Speaking on Deep Background, the Press Secretary whispered: Level 4 datacenters can, and will, fail. Nothing you can do including just doing 48V DC for everything are truly foolproof solutions. Hard to find anyone who takes the -48vdc mantra to heart more than an RBOC. Ditto on lightning protection. Yet I recall the Bell South 305-255 CO taking a lightning hit on the incoming power; the 5ESS was down for 3-4 hours. -- A host is a host from coast to [EMAIL PROTECTED] no one will talk to a host that's close[v].(301) 56-LINUX Unless the host (that isn't close).pob 1433 is busy, hung or dead20915-1433
Re: An Internet IPv6 Transition Plan
On Wed, Jul 25, 2007 at 07:52:19PM +0800, Adrian Chadd wrote: On Wed, Jul 25, 2007, Stephen Wilcox wrote: Lack of IPv4 addresses will put the brakes on growth of the Internet which will have a major impact on revenue growth. Before long stock market analysts are going to be asking tough questions, and CEOs are suddenly going to see the IPv6 light. What exactly will cease to grow tho? The 4 billion IPs that have always been around will continue to be. I think you overestimate the effects.. All the existing big businesses can operate with what they already have, Google and Yahoo are not going to face any sort of crisis for the foreseeable future. And as I've been saying for a while and Randy put in his presentation, supply and demand will simply cause the cost of having public IPs to go up from zero to something tiny - enough to see IPs being put back into the pool to those who really need them. I'm not sure what your definition of really tiny is, but out here IPs are a dollar or two each a year from APNIC. I'm sure ARIN's IP charges aren't $0.00. RIPE is a couple thousands Euros to be an LIR which gets you all the IPs you need.. $1/yr is like 8c/month - well into the realm of being sunk into the cost when you provide a hosting service or DSL line. Its close enough to zero to be lost in the overheads of any business operation. Now, if you suddenly charge $2.50/mo to have a public IP or $15/mo for a /28 it does become a consideration to the customer as to if they _REALLY_ need it Steve
Re: An Internet IPv6 Transition Plan
John, On Jul 25, 2007, at 1:14 PM, John Curran wrote: All the existing big businesses can operate with what they already have, Google and Yahoo are not going to face any sort of crisis for the foreseeable future. And as I've been saying for a while and Randy put in his presentation, supply and demand will simply cause the cost of having public IPs to go up from zero to something tiny - enough to see IPs being put back into the pool to those who really need them. Putting them back into circulation doesn't work unless its done in very large chucks to major ISPs. If this isn't the model followed, then we will see a lot more routes for the equivalent number of new customers. People complaining about the ability to carry both IPv6 and IPv4 routing need to think carefully about how long we'll actually last if the ISP's are injecting thousands of unaggregatable routes from recovered address space each day. Been there, done that, got several t-shirts. Longer prefixes _will_ hit the routing system. ISPs will react by (re-)implementing prefix length filters. Many people will whine. Additionally, the run rate for IPv4 usage approximates 10 /8 equivalents per year and increasing. Even given great legacy recovery, you've only gained a few more years and then still have to face the problem. This assumes consumption patterns remain the same which is, I believe, naive. In a world where you have to pay non-trivial amounts for address space utilization, people will only use the address space they actually need and you'll see even more proliferation of NAT for client-only services. Rgds, -drc
Where did freeipdb IP utility site go?
I was trying to investigate some the ip management tools and followed the link www.freeipdb.org and was more than a little upset with what I found. This domain name apparently has been taken by a porn site that is wanting to auction it off. does anyone know if the project died or if it changed domain names. I have removed the reference to it in the wiki page, but there are other references to the site on the NANOG site. I am not sure who will need to remove the links, but they no longer point to an ip management tool. If the utility still exist I would be intersted in finding it, as I saw not able to dig it up on a quick Google search. -- Brian Raaen Network Engineer [EMAIL PROTECTED]
Re: An Internet IPv6 Transition Plan
At 2:02 PM +0200 7/25/07, David Conrad wrote: This assumes consumption patterns remain the same which is, I believe, naive. In a world where you have to pay non-trivial amounts for address space utilization, people will only use the address space they actually need and you'll see even more proliferation of NAT for client-only services. I believe that we'll see extensive use of NAT for client-only services (just look at many broadband residential services today), but that won't help business customers who want a block for the DMZ servers. They'll pay, but the question is whether they can afford the actual global cost of routing table entry, or whether it will even be accountable. ISP's can figure out the cost of obtaining IPv4 blocks, but the imputed cost of injecting these random blocks into the DFZ routing table is harder to measure and inflicted on everyone else. /John
Re: An Internet IPv6 Transition Plan
At 1:15 PM +0100 7/25/07, Stephen Wilcox wrote: At present, there's a few years for these folks to switch to IPv6 for their growth. It requires cooperation from the Internet, in that we all need to recognize that there will be IPv6 customers out there soon, and even if you don't plan on having those, please make your public facing servers IPv6 reachable in the next few years. I'm not sure there is time for v6 to be ready before companies find different ways to manage this. There are many things that need to happen to enable v6 and I dont think any of them are happening in a big way. Whether the large CDNs deploy v6, if v6 can be purchased in volume as transit are likely to be the major factors.. Steve - Are you unable to make your public facing servers IPv6-reachable? /John
Re: An Internet IPv6 Transition Plan
On Wed, Jul 25, 2007 at 12:21:04PM +0100, [EMAIL PROTECTED] wrote: Lack of IPv4 addresses will put the brakes on growth of the Internet which will have a major impact on revenue growth. Before long stock market analysts are going to be asking tough questions, and CEOs are suddenly going to see the IPv6 light. What exactly will cease to grow tho? The 4 billion IPs that have always been around will continue to be. I think you overestimate the effects.. I think you misunderstand the dictionary definition of growth. Yes, the IPv4 addresses, and much of the network infrastructure using them, will continue to be. But growth is about expansion, adding more, increasing the size and scope of the network. Few businesses are satisfied with collecting the same monthly recurring revenue from the same customer base. They either want to grow the customer base or grow the monthly revenue per customer. In the Internet business the main engine of revenue growth is growing the customer base by growing the network and adding more customers. I dont think paypal's growth is tied to how many IPs they have... I think it relates to how many hits www.paypal.com receives and what their products look like. IP availability is unlikely to ever have more than the briefest mention in the boardroom and probably only in response to a news article quoting the end of the internet being imminent. And as I've been saying for a while and Randy put in his presentation, supply and demand will simply cause the cost of having public IPs to go up from zero to something tiny - enough to see IPs being put back into the pool to those who really need them. And when your Internet supplier tells you that there will be a $10 per month increase in fees to cover the increase cost of IPv4 addresses, will you be happy? Will you start shopping for an IPv6 Internet supplier? When IPv6 Internet access is cheaper due to IPv4 address costs, then ISPs face a wholesale loss of their customer base. Of course, most business managers are smart enough to see this coming and resist paying for IPv4 addresses in the first place. I'll sell you v6 today for 1/4 of the price of v4. Providing you understand theres not a lot out there. I agree on your cost comparison, but consider what investment and costs are needed to be able to get to that point. this model of business. When the IPv4 shortage begins to bite, then you will see enormous amounts of money and effort put into IPv6 conversions (and new IPv6 startups who intend to unseat Google, Yahoo, etc.). You will just see redeployment of existing budgets.. why would you pay more to see the same webpage be delivered just because of some techno mumbo jumbo Any investor would be crazy to invest in a v6 competitor to Google.. enter a mature market using a new technology that 99% of the planet cant get to? The only folks getting into v6 are the ones controlling the v4 market with enough spare RD cash currently. Steve
Re: An Internet IPv6 Transition Plan
On Wed, Jul 25, 2007 at 07:50:05AM -0400, John Curran wrote: At 12:30 PM +0100 7/25/07, Stephen Wilcox wrote: Hi John, I fully agree on that.. but I am disagreeing as to the timescales. There is some opinion that when IANA hands out the last of its IP blocks things will change overnight, and I dont see any reason for that to be the case. I think there are a lot of IPs currently allocated to ISPs but as yet unassigned to customers, and I think there will be a lot of policy changes to make more efficient use of the space that is already out there - I specifically think that will come from ISPs reusing IPs and setting costs that ensure they continually have IPs available to customers willing to pay for them. In the ARIN region, we've got major ISP's coming back every 6 months with high utilization rates seeking their next block to allow customer growth. While I'm certain that some internal recovery is possible, there's a realistic limit of how long any ISP can make their air supply last. I think the combined effect of these things means - we will not be running into a wall at any time - availability of IPs will slowly decrease over time (as cost slowly increases) - adoption of NAT and v6 will be an ongoing trend with no sudden increase Unless the policy changes you suggest somehow dramatically change the current usage rate, we're going to have a very serious rate of change when the IANA/RIR pool hits zero. That sort of defines hitting a wall, by my definition. Well, you already say you have major ISPs submitting requests every 6 months, and I guess that is your high water mark so everyone else should be longer (at lease here under RIPE you are supposed to be allocated space for 2 yrs at a time). So, we have IANA out of space at eof 2009.. that will then take the RIRs 12 to 24 mo to allocate that out before there is any impact on ISPs. Once that occurs we still have your 6mo-2yr+ period that ISPs have in their allocated and unused pool to be giving to customers. Add all that together and you have 18mo-4yrs of 'greyness', no overnight wall. And I'm saying each of the events plus that grey period will cause evolution in the market place to occur such that there are no walls or catastraphies from a continuity or economical point of view. Please propose the magical policy changes asap... we need to get them through the public process and adopted in record time to have any affect on the usage rate. Well, thats a different story. Inflating the price of IPs would have been a good thing but I think that horse has already bolted now. This means no end of the world as we know it, and no overnight adoption of new technology.. just business as usual in an evolving environment. Note: I'm not advocating an overnight technology deployment; just advising those folks who presently rely on continuous availability of new address blocks from the RIR's that we're going to see a change. Indeed they will, but it wont happen to everyone at the same time (as they all have months or years of IPs left) and they have plenty of time to figure out how to adapt their products and business models. At present, there's a few years for these folks to switch to IPv6 for their growth. It requires cooperation from the Internet, in that we all need to recognize that there will be IPv6 customers out there soon, and even if you don't plan on having those, please make your public facing servers IPv6 reachable in the next few years. I'm not sure there is time for v6 to be ready before companies find different ways to manage this. There are many things that need to happen to enable v6 and I dont think any of them are happening in a big way. Whether the large CDNs deploy v6, if v6 can be purchased in volume as transit are likely to be the major factors.. Steve
Re: An Internet IPv6 Transition Plan
On Wed, Jul 25, 2007 at 08:18:30AM -0400, John Curran wrote: At 1:15 PM +0100 7/25/07, Stephen Wilcox wrote: At present, there's a few years for these folks to switch to IPv6 for their growth. It requires cooperation from the Internet, in that we all need to recognize that there will be IPv6 customers out there soon, and even if you don't plan on having those, please make your public facing servers IPv6 reachable in the next few years. I'm not sure there is time for v6 to be ready before companies find different ways to manage this. There are many things that need to happen to enable v6 and I dont think any of them are happening in a big way. Whether the large CDNs deploy v6, if v6 can be purchased in volume as transit are likely to be the major factors.. Steve - Are you unable to make your public facing servers IPv6-reachable? Well, I wear a few hats these days :) but.. I think the short answer is yes, I'm unable. Most stuff I am involved in is modern enough that the servers have a v6 stack so that could be enabled. But the apps themselves are not all v6 so they would either need to be upgraded or fixed. We would of course need to configure these and ensure all dependncies are v6 capable, particularly if we're sending address info back to customers we dont want to switch them in and out of v4/v6. Then the network gear tends to be v6 enabled in the core and not at the edges where older gear has been redeployed. And a lot of the gear that claims to be v6 doesnt handle hardware switching properly so that needs investigating and would be an issue. Then we'd need to make sure all security and policies are uniform and working equally across v6. Assuming we sort it tho then we need to bring up v6 transit, more v6 peers and drop any v4 tunnels as they cant be expected to handle production load. I guess theres abstraction to fix too - my CMS, monitoring, allocation, much of which is automated and all of which relies on storing address info would all need to be rewritten to allow v6 addresses on hosts, interfaces, customers etc So fix all that and yes we could have v6 servers, but you also said reachable and according to my BGPv6 table theres very little reachable out there right now - about 700 prefixes when compared to 25000 v4 ASNs that should each be visible. So you can break this into two elements - stuff I control and stuff I dont. For the stuff I control I think the summary is that I'd need to build an ISP from scratch essentially (if not in terms of capex purchases then certainly in terms of design and implementation). And the stuff I dont control, well.. I cant do much about that. Steve
Re: San Francisco Power Outage
On Tue, Jul 24, 2007 at 09:57:09PM -0500, Brandon Galbraith wrote: It appears that 365 is using the Hytec Continuous Power System [ http://hitec.pageprocessor.nl/p3.php?RubriekID=2016], which is a motor, generator, flywheel, clutch, and Diesel engine all on the same shaft. They don't use batteries. Yes. I used to work for the company that originally built the 365 Main datacenter and remember touring it near the end of the construction phase. The collection of power units up on the roof was impressive, as were the seismic isolators in the basement. But even when you try and do everythying right Murphy usually finds a way to sneak up behind you and whisper BOHICA in your ear. For example, we had a failure at another datacenter that uses Piller units, which operate on the same basic principle as the Hitec ones. While running on generator one of the engines overheated due to an oil-flow problem and threw a rod. When the on-duty electrician responded to the alarm, there were red-hot chunks of engine *outside* of the enclosure, and there was a hole in the side of the unit large enough to stick your arm in. The facility manager kept the damaged piston as a momento. :-) I don't remember whether this was due to a design flaw, improper installation, or what, but the important points are that (1) this is the real world and shit happens, and (2) it wasn't until the generator was worked long enough that the reduction in oil flow caused enough friction to trigger a catastrophic failure. I.e., there's no guarantee that you will catch this kind of problem in your monthly tests. On Tue, Jul 24, 2007 at 05:39:34PM -0700, George William Herbert wrote: Unfortunate real-world lesson: there is a functional difference between pushing the UPS test cutover button, and some of the stuff that can happen out on the power lines (including rapid voltage swings, harmonics, etc). Precisely. --Jeff
Re: An Internet IPv6 Transition Plan
At 1:15 PM +0100 7/25/07, Stephen Wilcox wrote: I'm not sure there is time for v6 to be ready before companies find different ways to manage this. There are many things that need to happen to enable v6 and I dont think any of them are happening in a big way. Let's agree on 18mo-4yrs of 'greyness' (as you put it), and that indeed different companies find different ways to manage this... Some of the companies are going to select IPv6 because it's has some level of support in existing end systems and network gear (even considering the various implementation flaws, lack of hardware support, etc), and because it supports a generally hierarchical addressing/routing model which works (again, despite recognition that the routing system has some serious long-term scalability questions which need to be looked into). For their choice to work, it's necessary that your public-facing servers accept IPv6 connections. It's really not a hard concept, and it's based on the simple premise stated by Jon: In general, an implementation should be conservative in its sending behavior, and liberal in its receiving behavior. You've stated a long list of items that need to be changed, but that's if you want to serve as an ISP using IPv6 for customers, and change your internal infrastructure to IPv6, and that's not required. You've already said you are going to take another path to manage things, and that's cool. The question is whether you still recognize the need to deploy IPv6 on the very edge of your network for your public services such as web and email. You could even have someone host this for you, it's not that hard, and there's two to 4 years to get it done. If you're saying that no one at all needs to use IPv6, so you aren't going to worry about IPv6 connectivity for your public facing servers, then it would be best to explain how global routing is supposed to work when ISP's aren't using predominantly hierarchical address assignments for their growth. /John
Re: An Internet IPv6 Transition Plan
John, On Jul 25, 2007, at 2:13 PM, John Curran wrote: I believe that we'll see extensive use of NAT for client-only services (just look at many broadband residential services today), but that won't help business customers who want a block for the DMZ servers. Well yes. However there are likely to be far fewer devices in the DMZ that need numbers. In addition, renumbering DMZ servers is a whole lot less painful than renumbering your entire network, so perhaps PA space would be more acceptable. I can easily imagine a world where ISPs migrate their internal infrastructure that is currently numbering in IPv4 space over to IPv6, thereby freeing up a large amount of IPv4 space that could then be used for customer DMZ servers. My point is that once you associate a non-trivial cost per address, people will tend to use address space more efficiently (either by reusing space more efficiently or reducing the amount of space they need). As such address consumption rates will change. They'll pay, but the question is whether they can afford the actual global cost of routing table entry, or whether it will even be accountable. It never has been. Not sure why this would change. As we've seen in the past, it's much easier to do prefix length filters when it becomes an issue. ISP's can figure out the cost of obtaining IPv4 blocks, but the imputed cost of injecting these random blocks into the DFZ routing table is harder to measure and inflicted on everyone else. http://en.wikipedia.org/wiki/Tragedy_of_the_commons Rgds, -drc
Re: Where did freeipdb IP utility site go?
Dear Brian, I was trying to investigate some the ip management tools and followed the link www.freeipdb.org and was more than a little upset with what I found. This domain name apparently has been taken by a porn site that is wanting to auction it off. does anyone know if the project died or if it changed domain names. I have removed the reference to it in the wiki page, but there are other references to the site on the NANOG site. I am not sure who will need to remove the links, but they no longer point to an ip management tool. If the utility still exist I would be intersted in finding it, as I saw not able to dig it up on a quick Google search. also quick google search: http://www.rickyninja.net/ could also be interesting: http://www.brownkid.net/NorthStar bye, Ingo
RE: iPhone and Network Disruptions ...
On Tue, 24 Jul 2007, Frank Bulk wrote: If you look at Kevin's example traces on the EDUCAUSE WIRELESS-LAN listserv you'll see that the ARP packets are in fact unicast. Iljitsch's point about the fact that iPhones remain on while crossing wireless switch boundaries is exactly dead on. If you read the security advisory you'll see that it involves either L3 roaming or two or more WLCs that share a common L2 network. Most wireless clients don't roam in such a big way. With the exception of our 1000+ Cisco 7920 phones... Then again, they probably work just fine with Cisco's other products, heh. - d. -- Dominic J. Eidson Baruk Khazad! Khazad ai-menu! - Gimli http://www.the-infinite.org/
Re: An Internet IPv6 Transition Plan
On 25 Jul 2007, at 14:15, Stephen Wilcox wrote: [...] Well, you already say you have major ISPs submitting requests every 6 months, and I guess that is your high water mark so everyone else should be longer (at lease here under RIPE you are supposed to be allocated space for 2 yrs at a time). A recent policy change means that The RIPE NCC allocates enough address space to LIRs to meet their needs for a period of up to 12 months. http://www.ripe.net/ripe/docs/ipv4-policies.html#5 So, we have IANA out of space at eof 2009.. that will then take the RIRs 12 to 24 mo to allocate that out before there is any impact on ISPs. If there isn't a run on the bank. Leo
Re: 365 Main - an operators' nightmare?
Jason J. W. Williams wrote: I believe this happened to an Internap facility in Seattle a couple of years ago: http://community.livejournal.com/lj_dev/670215.html I was told it happened in our colo facility about a month before we moved in. Some unfortunate remodeling of previous data center space had left an EPO switch in a janitor's closet. The maid knocked loose the protective covering, which of course made an alarm start screaming...so she hit the EPO to stop the noise. Did it work? (Did that stop the noise, things got real quiet?) -- Jay Hennigan - CCIE #7880 - Network Engineering - [EMAIL PROTECTED] Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 - WB6RDV
RE: Where did freeipdb IP utility site go?
Don't know about freeipdb, but we use IPPlan: http://iptrack.sourceforge.net/ -Scott -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Brian Raaen Sent: Wednesday, July 25, 2007 8:13 AM To: nanog@merit.edu Subject: Where did freeipdb IP utility site go? I was trying to investigate some the ip management tools and followed the link www.freeipdb.org and was more than a little upset with what I found. This domain name apparently has been taken by a porn site that is wanting to auction it off. does anyone know if the project died or if it changed domain names. I have removed the reference to it in the wiki page, but there are other references to the site on the NANOG site. I am not sure who will need to remove the links, but they no longer point to an ip management tool. If the utility still exist I would be intersted in finding it, as I saw not able to dig it up on a quick Google search. -- Brian Raaen Network Engineer [EMAIL PROTECTED]
Re: An Internet IPv6 Transition Plan
Hi Stephen, I have run many times in the kind of problems that you describe, and always was able to find a suitable alternative solution, at least a temporary one (for instance until specific hardware can be upgrades, such as L3 switches, and the solution was working fine at least for initial small IPv6 traffic). For example, I've been able to use with IPv6 many applications that don't support it, but means of using portproxy. I'm probably able to help you (and/or other folks) with more specific examples, so if you're interested, write me offline. Regards, Jordi De: Stephen Wilcox [EMAIL PROTECTED] Responder a: [EMAIL PROTECTED] Fecha: Wed, 25 Jul 2007 13:41:57 +0100 Para: John Curran [EMAIL PROTECTED] CC: nanog@merit.edu Asunto: Re: An Internet IPv6 Transition Plan On Wed, Jul 25, 2007 at 08:18:30AM -0400, John Curran wrote: At 1:15 PM +0100 7/25/07, Stephen Wilcox wrote: At present, there's a few years for these folks to switch to IPv6 for their growth. It requires cooperation from the Internet, in that we all need to recognize that there will be IPv6 customers out there soon, and even if you don't plan on having those, please make your public facing servers IPv6 reachable in the next few years. I'm not sure there is time for v6 to be ready before companies find different ways to manage this. There are many things that need to happen to enable v6 and I dont think any of them are happening in a big way. Whether the large CDNs deploy v6, if v6 can be purchased in volume as transit are likely to be the major factors.. Steve - Are you unable to make your public facing servers IPv6-reachable? Well, I wear a few hats these days :) but.. I think the short answer is yes, I'm unable. Most stuff I am involved in is modern enough that the servers have a v6 stack so that could be enabled. But the apps themselves are not all v6 so they would either need to be upgraded or fixed. We would of course need to configure these and ensure all dependncies are v6 capable, particularly if we're sending address info back to customers we dont want to switch them in and out of v4/v6. Then the network gear tends to be v6 enabled in the core and not at the edges where older gear has been redeployed. And a lot of the gear that claims to be v6 doesnt handle hardware switching properly so that needs investigating and would be an issue. Then we'd need to make sure all security and policies are uniform and working equally across v6. Assuming we sort it tho then we need to bring up v6 transit, more v6 peers and drop any v4 tunnels as they cant be expected to handle production load. I guess theres abstraction to fix too - my CMS, monitoring, allocation, much of which is automated and all of which relies on storing address info would all need to be rewritten to allow v6 addresses on hosts, interfaces, customers etc So fix all that and yes we could have v6 servers, but you also said reachable and according to my BGPv6 table theres very little reachable out there right now - about 700 prefixes when compared to 25000 v4 ASNs that should each be visible. So you can break this into two elements - stuff I control and stuff I dont. For the stuff I control I think the summary is that I'd need to build an ISP from scratch essentially (if not in terms of capex purchases then certainly in terms of design and implementation). And the stuff I dont control, well.. I cant do much about that. Steve ** The IPv6 Portal: http://www.ipv6tf.org Bye 6Bone. Hi, IPv6 ! http://www.ipv6day.org This electronic message contains information which may be privileged or confidential. The information is intended to be for the use of the individual(s) named above. If you are not the intended recipient be aware that any disclosure, copying, distribution or use of the contents of this information, including attached files, is prohibited.
Re: DNS Hijacking by Cox
Mattias Ahnberg wrote: Peter Dambier wrote: The problem is, you dont know what is behind that probably NATted ip address. Probably you have 3 unix machines running smtp and uucp and a single infected windows box and maybe some VoIPs and ... This is why I spoke of merely intercepting web traffic to inform, to not interrupt other services that may use the same link. I am in the same situation myself, sharing lots of stuff via the same fiber to my house. I even have TV through it. So I actually thought of that. You are right. Intercepting is mostly harmless. And an ISP probably knows a bit more about their customer base than what we do, so this idea would ofcourse have to adapt to that. But as said, its a complicated matter and probably not a good idea either way before we know who is supposed to do what and for whom. Having been a costumer to some ISPs and communicating with others, I dont agree. At least concerning email they dont have a clue about their costumers and there are others things like uucp, VoIP and p2p or IPv6 tunnels they dont have either. Kind regards Peter and Karin -- Peter and Karin Dambier Cesidian Root - Radice Cesidiana Rimbacher Strasse 16 D-69509 Moerlenbach-Bonsweiher +49(6209)795-816 (Telekom) +49(6252)750-308 (VoIP: sipgate.de) mail: [EMAIL PROTECTED] mail: [EMAIL PROTECTED] http://iason.site.voila.fr/ https://sourceforge.net/projects/iason/ http://www.cesidianroot.com/
Re: An Internet IPv6 Transition Plan
Thus spake Adrian Chadd [EMAIL PROTECTED] I'm not sure what your definition of really tiny is, but out here IPs are a dollar or two each a year from APNIC. I'm sure ARIN's IP charges aren't $0.00. The 73 Xtra Large LIRs that consume 79% of ARIN's v4 space today are paying no more than USD 0.03 per IP per year. That's not quite zero, but it's close enough the effect is the same. Until the cost of v4 space to these folks is more than a rounding error, they have absolutely no incentive to conserve. It doesn't matter what the other 2550 LIRs do because they're insignificant factors in overall consumption. S Stephen Sprunk Those people who think they know everything CCIE #3723 are a great annoyance to those of us who do. K5SSS --Isaac Asimov
Why do we use facilities with EPO's?
I was complaining to some of the power designers during the building of a major facility that the EPO button represented a single point of failure, and effectively made all of the redundancy built into the power system useless. After all, what's the point of having two (or more) of anything, if there's one button somewhere that turns it all off? What I found interesting is that a single EPO is not a hard and fast rule. They walked me through a twisty maze of the national electric code, the national fire code, and local regulations. Through that journey, they left me with a rather interesting tidbit. The more urban an area the more likely it is to have strict fire codes. Typically these codes require a single EPO for the entire structure, there's no way to compartmentalize to rooms or subsystems. However in more rural areas this is often not so, and they had in fact built data centers to code WITHOUT a single building EPO in several locations. That's to say there was no EPO, but that it may only affect a single room, or even a single device. If they can be avoided, why do we put up with them? Do we really want our colo in downtown San Francisco bad enough to take the risk of having a single point of failure? How can we, as engineers, ask questions about how many generators, how much fuel, and yet take for granted that there is one button on the wall that makes it all turn off? Is it simply that having colo in the middle of the city is so convenient that it overrides the increased cost and the reduced redundancy that are necessitated by that location? -- Leo Bicknell - [EMAIL PROTECTED] - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ Read TMBG List - [EMAIL PROTECTED], www.tmbg.org pgpLvtSetOd1U.pgp Description: PGP signature
Re: Why do we use facilities with EPO's?
On Wed, 25 Jul 2007, Leo Bicknell wrote: What I found interesting is that a single EPO is not a hard and fast rule. They walked me through a twisty maze of the national electric code, the national fire code, and local regulations. Through that journey, they left me with a rather interesting tidbit. Well. An Emergency Power Off button is a NEC (the electrical code adopted in most of the USA) trade-off for allowing more flexible wiring practices in computer rooms. If you don't use any of those alternative wiring practices, you aren't required to install an EPO (modulo the rare local code variation). The problem is some people want to get rid of the EPO, but also want to keep using alternative wiring practices. If you've looked in many computer rooms, you'll see some creative wiring practices in use. You'll notice Telco central offices don't have building EPOs. Likewise Equinix data centers don't have EPOs. But they have limits on what wiring practices can be used in their facilities compared to other data centers.
Re: An Internet IPv6 Transition Plan
On Tue, Jul 24, 2007 at 10:01:44AM -0400, Chad Oleary wrote: DHCPv6 doesn't even hand out addresses. I wasn't going to say anything because Alain already said something. But we've gotten this question from at least two other sources in the last two days who read this and wanted to ask us what that was about. What were they thinking? It does seem pretty weird. So hopefully it will help people who don't have a geek to ask if I were to explain what's going on here: There are 'stateless' and 'stateful' ways to implement DHCPv6. You don't get address assignment unless you do 'stateful' DHCPv6 (and then it's complicated by wether you mean 'normal' addresses, 'temporary' addresses which change every renew, or 'prefix delegation'). But DHCPv6 does give out addresses. The easy way to think of DHCPv6 stateful vs stateless is to realize we have the same relationship in DHCPv4 - you can get an address like people normally do with DHCPv4, or you can use a DHCPINFORM if you already have one...so you can get configuration values like nameservers and such without allocating an address. That's all stateless DHCPv6 is. What Alain said is that until 12-18 months prior to today, there have not been very many sources of stateful DHCPv6 implementations. There are several implementations out now, many appearing enabled by default on production software you probably already have in your networks. -- Ash bugud-gul durbatuluk agh burzum-ishi krimpatul. Why settle for the lesser evil? https://secure.isc.org/store/t-shirt/ -- David W. HankinsIf you don't do it right the first time, Software Engineeryou'll just have to do it again. Internet Systems Consortium, Inc. -- Jack T. Hankins pgpjg14h4FrXY.pgp Description: PGP signature
TWTC issue with Foundry routers?
Anyone know of any changes that were made with TWTC (AS 4323) last night that may have affected those running Foundry routers? We peer with a number of providers and last night our TWTC connection went down with: Jul 25 15:57:22:N:BGP Peer 1.2.3.49 DOWN (Attribute Flags Error) Jul 25 15:57:14:N:BGP Peer 1.2.3.49 UP (ESTABLISHED) If I debug updates on that session I get: (Lines added for readability) Jul 25 15:57:21 BGP: 1.2.3.49 rcv UPDATE 142.166.102.0/24 Jul 25 15:57:21 BGP: 1.2.3.49 rcv UPDATE w/attr: Origin=IGP AS_PATH=AS_SEQ(2) 4323 8881 8881 8881 30915 NextHop=1.2.3.49 COMMUNITY=4323:51 4323:501 4323:1003 4323:2001 4323:2503 4323:34510 4323:5 65101:1003 65102:4 65103:1 65104:301 Jul 25 15:57:21 BGP: 1.2.3.49 rcv UPDATE 193.27.220.0/23 Jul 25 15:57:21 BGP: 1.2.3.49 rcv UPDATE w/attr: Origin=IGP AS_PATH=AS_SEQ(2) 4323 2828 19092 14188 14188 14188 14188 14188 NextHop=1.2.3.49 COMMUNITY=4323:51 4323:501 4323:1015 4323:2503 4323:36410 4323:5 65101:1015 65102:4 65103:1 65104:301 Jul 25 15:57:21 BGP: 1.2.3.49 rcv UPDATE 64.13.0.0/22 Jul 25 15:57:21 BGP: 1.2.3.49 rcv invalid COMMUNITY attribute flag d0 Jul 25 15:57:21 BGP: 1.2.3.49 rcv UPDATE w/attr: Origin=IGP AS_PATH=AS_SEQ(2) 4323 12956 3352 NextHop=1.2.3.49 ATOMIC_AGGREGATE AGGREGATOR AS=3352 Speaker=81.46.63.133 The router is a Foundry NetIron 400 running their 7.8 code. We have two of these talking to Level 3, TWTC, Cogent, Uunet and ATT and only the TWTC had an issue. They sent me a default route instead of full routes and the session came up and was stable; go back to full routes and error. They admitted to me this afternoon that three other customers are having the same issue. That's when we started wondering if they changed something that the Foundry code doesn't like. Interesting though is that they claim to not be sending me communities while the output above indicates they are. Any ideas; be nice to get the link back up. :-) Thanks, David
Re: 365 Main - an operators' nightmare?
Or: So I'm working at this place that is really cheap... Our CTO believes that it is stupid to pay for electricians that have experience working in datacenters, because after all, power is power, right? So, he calls a bunch of people in the Yellow Pages and hires the cheapest guy he can find. Said person arrives and looks a little goggle eyed at all the power stuff -- I wander back in a few hours later and he is sitting in the middle of the floor reading the Users Manual for the UPS.. Anyway, he manages to run the three new circuits for us without killing himself (although for some reason keeps switching the UPS between online and bypass), and then starts walking out the door... He stops at the door, looks at the big red glowing switch marked Emergency Power Off -- and then pushes it. Everything goes quiet, apart from Rob got startled and dropped the shelf he was mounting onto his foot. After we got things turned back on we ask the electrician what exactly he was thinking... Well, I figured the light was on because you were running on Emergency Power... W I believe this happened to an Internap facility in Seattle a couple of years ago: http://community.livejournal.com/lj_dev/670215.html I was told it happened in our colo facility about a month before we moved in. Some unfortunate remodeling of previous data center space had left an EPO switch in a janitor's closet. The maid knocked loose the protective covering, which of course made an alarm start screaming...so she hit the EPO to stop the noise. Thankfully, the switch has been since removed... Anyhow, any story involving an EPO at 365 Main seems plausible... -J -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jim Popovitch Sent: Tuesday, July 24, 2007 8:59 PM To: Rusty Hodge Cc: [EMAIL PROTECTED] Edu Subject: Re: 365 Main - an operators' nightmare? On Tue, 2007-07-24 at 19:26 -0700, Rusty Hodge wrote: Think that's good? It gets better http://valleywag.com/tech/breaking/angry-mob-gathers-outside-sf- datacenter-282053.php That article states that only Colo 4 was affected. I'm in Colo 7 and it was affected as well. You're not seriously believing the disgruntled employee story are you? No. ;-) But it is otherwise believable. I've seen people hit big-red-buttons in disbelief before, doing so in anger seems very plausible. -Jim p. !SIG:46a6d6e0156535690315935! -- It's a mistake trying to cheer up camels. You might as well drop meringues into a black hole. -- Terry Prachett
Re: Why do we use facilities with EPO's?
Leo Bicknell wrote: I was complaining to some of the power designers during the building of a major facility that the EPO button represented a single point of failure, and effectively made all of the redundancy built into the power system useless. After all, what's the point of having two (or more) of anything, if there's one button somewhere that turns it all off? If the EPO cuts power to the whole facility, it didn't fail, it worked perfectly. The failure is actually Slappy the EPO button-pusher. ~Seth
Re: Why do we use facilities with EPO's?
A few weeks ago a rural farmer in southern Maryland succumbed to methane gas in a manure pile. His wife went to rescue him, and she too collapsed. Their two daughters followed. All 4 died. And that is why we have EPO's; to keep the rescuers from become cascade victims. -- A host is a host from coast to [EMAIL PROTECTED] no one will talk to a host that's close[v].(301) 56-LINUX Unless the host (that isn't close).pob 1433 is busy, hung or dead20915-1433
Re: Why do we use facilities with EPO's?
John C. A. Bambenek wrote: Funny story about that and the EPO we have here... ... Story #1 Many years ago, the safety department for my employer made a big stink over the fact that the EPO hadn't been tested in a couple of years. We scheduled an outage window, shut everything down. The facilities guy pressed the magic big RED button and NOTHING! Tracing the problem back, there was a blown fuse in the EPO circuit because a wire had shorted. A real safe design! Story #2 Every few years the EPO buttons would change. First they were the ones with the metal ring around the button that protects against accidental pushing. Then we would get the mushroom button because it was safer. Invariably someone would trip it and they would change them back. I think some guy made some money submitting suggestions to change the button every few years.
EPO/NEC (was Re: Why do we use facilities with EPO's?)
On Wed, 25 Jul 2007, Leo Bicknell wrote: What I found interesting is that a single EPO is not a hard and fast rule. They walked me through a twisty maze of the national electric code, the national fire code, and local regulations. Through that journey, they left me with a rather interesting tidbit. The more urban an area the more likely it is to have strict fire codes. Typically these codes require a single EPO for the entire structure, there's no way to compartmentalize to rooms or subsystems. However in more rural areas this is often not so, and they had in fact built data centers to code WITHOUT a single building EPO in several locations. That's to say there was no EPO, but that it may only affect a single room, or even a single device. If they can be avoided, why do we put up with them? Do we really want our colo in downtown San Francisco bad enough to take the risk of having a single point of failure? How can we, as engineers, ask questions about how many generators, how much fuel, and yet take for granted that there is one button on the wall that makes it all turn off? Is it simply that having colo in the middle of the city is so convenient that it overrides the increased cost and the reduced redundancy that are necessitated by that location? This is an interesting question. National Electric Code (NEC) requires EPO. Sort of. Articles 645 and 685 deal with it. While NEC is not binding on every jurisdiction, almost every US jurisdiction bases its code on NEC with additions/subtractions. I don't know offhand if the local changes deal with EPO much, however, here's some food for thought regarding EPO and NEC. With regard to putting up with them - EPOs are designed to protect life, not property or uptime. If there's a short causing electrical fire because breaker did not open, firefighter better be sure he can cut the power *before* stepping next to it. Here's how NEC works: 1) If a room is designed to comply with Article 645, it must have EPO, *except* if it qualifies under Article 685. Being under Article 645 gives couple of things that are generally not permitted otherwise, as follows: 645.4 D) permits underfloor wiring for power, receptacles and crossconnects. 645.4 E) Power cables; comunications cables; connecting cables; interconnecting cables; and associated boxes, connectors plugs and receptacles that are listed as part of, or for, information technology equipment shall not be required to be secured in place. In other words, you can have crossconnects that are laying on the floor (or under raised floor but not otherwise secured), and that is OK, normally they'd need to be secured every X feet. 645.17) (too lazy to retype NEC language) You can have PDUs with multiple panelboards within a single cabinet - not all that clear what exactly does it permit (PDUs with multiple breaker panels essentially). My understanding is that if you are willing to forego things that Article 645 permits, you do not have to install EPO. Frankly, I don't see all that much logic in 645 requirements and linking it to EPO (except, possibly, to make operation of datacenters not in compliance with 645 to be annoying enough that everyone would opt to comply with EPO). The Article 685 exception from EPO applies if An orderly shutdown is required to minimize personnel hazard and equipment damage. It is really intented for industrial (like chemical plants control) systems where EPO shutoff can cause damage to life/property. I doubt this applies to datacenter. Above is an armchair engineer's understanding. To be sure, you should consult a real engineer who can stamp and seal your plans! -alex
Re: Why do we use facilities with EPO's?
If they can be avoided, why do we put up with them? Major electrical fire, or arcing in the datacenter. Flood in the datacenter. Accidental water sprinkler discharge in the datacenter. Equipment fire that the FM-200 didn't put out, and you want not to have the sprinklers go off if you can avoid it. Equipment fire in a room without FM-200 in the first place. Equipment fire with sprinkler discharge, and you'd really rather just dry out the equipment than have to replace it all. Electrical worker who's electrocuted himself and going to die if you can't make the power go away. There are a very few exceptions, but for our practical purposes, people really ought to simply go to multiple site redundancy rather than thinking about bending major safety assumptions in how we operate individual buildings. You may have a few less outages, but you may also kill someone. What the Navy does on ships, for critical ship safety and combat systems; what the FAA does for their radar facilities and air traffic control facilities; what Telcos do, these are different operating regimes, and there are associated higher risk acceptance with the different equipment setups and safety procedures. -george william herbert [EMAIL PROTECTED]
Re: Why do we use facilities with EPO's?
I've always wondered who died or was injured and caused the EPO to come in to existence. There have been lots of EPO caused downtime stories, but does anyone on the NANOG list even have one single Thank God for the EPO story? I'll feel better about the general state of the world if I know that the EPO actually has a real valid use that has been ACTUALLY PROVEN IN PRACTICE rather than just in someone's mind. -Jerry Is so anti EPO, he has no remote EPO buttons, and even has the irrational fear about the jumper on the EPO terminal strip inside his UPSes coming undone.
Re: Why do we use facilities with EPO's?
I've always wondered who died or was injured and caused the EPO to come in to existence. There have been lots of EPO caused downtime stories, but does anyone on the NANOG list even have one single Thank God for the EPO story? I'll feel better about the general state of the world if I know that the EPO actually has a real valid use that has been ACTUALLY PROVEN IN PRACTICE rather than just in someone's mind. -Jerry Is so anti EPO, he has no remote EPO buttons, and even has the irrational fear about the jumper on the EPO terminal strip inside his UPSes coming undone. A friend of mine is a volunteer firefighter (and ex-ISP CEO, used to be on the list). I'm not sure that he'd voluntarily enter a burning datacenter to put it out if there wasn't an EPO. Firefighters won't use water on live electrical fires. If your response plan to a fire in the datacenter is the building burns down and hope nobody's inside it still then you're set. I've seen someone electrocuted and frozen on the wire in a non-datacenter setting; we flipped the building breaker, which was further than an EPO would have been. It wasn't through his chest, and was only 110 V, so it probably didn't make a difference that it took a minute to turn it off instead of 10 seconds. There were no severe burns, etc. I've seen equipment catch on fire in a datacenter. If I hadn't been able to cut off the power locally, the EPO was the last line of defense... People I know have hit the EPO when sprinklers discharged in the datacenter. If you're lucky these won't happen to you. But that's not why safety rules are put in place. Unluck happens. -george william herbert [EMAIL PROTECTED]
Re: Why do we use facilities with EPO's?
Speaking on Deep Background, the Press Secretary whispered:
{good examples deleted...}
There are a very few exceptions, but for our practical
purposes, people really ought to simply go to multiple
site redundancy rather than thinking about bending
major safety assumptions in how we operate individual
buildings. You may have a few less outages, but you
may also kill someone.
What the Navy does on ships, for critical ship safety
and combat systems; what the FAA does for their radar
facilities and air traffic control facilities; what Telcos
do, these are different operating regimes, and there are
associated higher risk acceptance with the different equipment
setups and safety procedures.
Indeed. The Navy uses a 'battleshort' at times. This is the
ultimate penny in the fusebox; a shunt around the breakers
on critical systems... such as main gun turret hydraulics
firing controls.
They are there because the analysis was it better to have some
things catch fire than to be impotent when the Bismark or the
Graf Spay is shooting at you.
[NASA used a similar battleshort at Houston MSC during the final
seconds of the A-11 descent, and other short critical periods.]
In both cases, HIGHLY trained people with LOTS of resources were
on the job. Plus, at least in the Navy case, casualties were an
accepted risk; even if you kill folks in one turret, if you get
a hit that saves your ship.
Needless to say, your insurance company is not likely take
that view, no matter how many pron feeds SLA's your EPO will
break. Nor will the local prosecutor trying a manslaughter case
when a fire-fighter dies.
--
A host is a host from coast to [EMAIL PROTECTED]
no one will talk to a host that's close[v].(301) 56-LINUX
Unless the host (that isn't close).pob 1433
is busy, hung or dead20915-1433
Re: iPhone and Network Disruptions ...
On Jul 24, 2007, at 5:34 PM, Iljitsch van Beijnum wrote: On 24-jul-2007, at 15:27, Prof. Robert Mathews (OSIA) wrote: Looking at this issue with an 'interoperability lens,' I remain puzzled by a personal observation that at least in the publicized case of Duke University's Wi-Fi net being effected, the ARP storms did not negatively impact network operations UNTIL the presence of iPhones on campus. The nagging point in my mind therefore, is: why have other Wi-Fi devices (laptops, HPCs/PDAs, Smartphones etc.,) NOT caused the 'type' of ARP flooding, which was made visible in Duke's Wi-Fi environment? Reading the Cisco document the conclusion seems obvious: the iPhone implements RFC 4436 unicast ARP packets which cause the problem. I don't have an iPhone on hand to test this and make sure, though. The difference between an iPhone and other devices (running Mac OS X?) that do the same thing would be that an iPhone is online while the user moves around, while laptops are generally put to sleep prior to moving around. There is also the weird property of many types of flood vulnerable systems that they seem to remain stable until some sort of threshold is reached before suddenly spiraling out of control. I am not sure of the exact mechanism behind this, but I have seen multiple instances of this happening. The standard scenario is basically: You have a couple of switches with STP turned off -- someone plugs in some random cable, forming a bridge loop... and everything continues running fine, until some time in the future when it all goes to hell in a hand-basket. Now, I could understand the system remaining stable until the first broadcast / unknown MAC caused flooding to happen, but I have seen this system remain stable for anywhere from a few days to in a few weeks before suddenly exploding. I have seen the same thing happen in systems other than switches, for example RIP networks with split-horizon turned off, weird frame-relay networks, etc. Unfortunately I have never managed to recreate the event in a controlled environment (In the few cases that I have cared enough to try, I form a loop and everything goes BOOM immediately!), and in the wild have always just fixed it and run away (its usually someone else's network and I'm just helping out or visiting or something). I HATE switched networks. A few observations: In *almost* all of the cases, things *do* go boom immediately! In the instances where they don't, there doesn't seem to be a correlation between load and when it does suddenly spiral out of control [0]. There is not a gradual increase increase in the sorts of packets that you would expect to see cause this (in a switched environment, you do not see flooded packets slowly increase, or even an exponential increase over a long time, there is basically no traffic and then boom! 100%). Anyway, I have wondered that triggers it, but never enough to actually look into much W [0] Except for one case that I remember especially fondly -- it was switched network with something like 30 switches scattered around -- someone had plugged one of those silver satin phone type cables (untwisted copper) between two ports on a switch -- the cable was bad enough that most of the frames were dropped / corrupted, but under high broadcast traffic loads enough packets would make it through to cause a flood, and then after some time (5-10 minutes) it would die back down... -- Never criticize a man till you've walked a mile in his shoes. Then if he didn't like what you've said, he's a mile away and barefoot.
Re: Why do we use facilities with EPO's?
On Jul 25, 2007, at 3:35 PM, Patrick W. Gilmore wrote: On Jul 25, 2007, at 2:03 PM, Tuc at T-B-O-H.NET wrote: If they can be avoided, why do we put up with them? Do we really want our colo in downtown San Francisco bad enough to take the risk of having a single point of failure? How can we, as engineers, ask questions about how many generators, how much fuel, and yet take for granted that there is one button on the wall that makes it all turn off? Is it simply that having colo in the middle of the city is so convenient that it overrides the increased cost and the reduced redundancy that are necessitated by that location? You forgot the default Single Point of Failure in anything.. HUMANS. The earth is a SPoF. Let's put DCs on the moon. Besides, safety always overrides convenience. And I don't think that is a bad trade off. Me neither... Having multiple redundant sites (and a well designed network between them) is almost always going to be better than a single, wildly redundant site. No matter how much redundancy you build into a single site, you cannot (realistically) engineer away things like floods, etc. Planning your redundancy and testing it though is very important... Random anecdote (from a friend, I don't know if it true or not): Back in the day (before cheap international circuits), a very large financial in New York needed connectivity to some branches in Europe, so they bought some capacity on a satellite transponder and built their own ground-station (not cheap) fairly close to NY. They then realized that the needed a redundant ground station in case the first one failed or something similar, so the built a second ground- station, just outside Jersey City One of the satellite connectivity failure modes is... rain fade. W -- TTFN, patrick -- Does Emacs have the Buddha nature? Why not? It has bloody well everything else!
Re: San Francisco Power Outage
[EMAIL PROTECTED] (Jonathan Lassoff) writes: Well, the fact still remains that operating a datacenter smack-dab in the center of some of the most inflated real estate in recent history is quite a castly endeavor. yes. (speaking for both 365 main, and 529 bryant.) I really wouldn't be all that surprised if 365 Main cut some corners here and there behind the scenes to save costs while saving face. no expense was spared in the conversion of this tank turret factory into a modern data center. if there was a dark start option, MFN ordered it. (but if it required maintainance, MFN's bankruptcy interrupted that, but the current owner has never been bankrupt.) As it is, they don't have remotely enough power to fill that facility to capacity, and they've suffered some pretty nasty outages in the recent past. I'm strongly considering the possibility of completely moving out of there. 2mW/floor seemed like a lot at the time. ~6kW/rack wasn't contemplated. (is it time to build out the land adjacent to 200 paul, then?) -- Paul Vixie
Re: Why do we use facilities with EPO's?
If they can be avoided, why do we put up with them? Do we really want our colo in downtown San Francisco bad enough to take the risk of having a single point of failure? How can we, as engineers, ask questions about how many generators, how much fuel, and yet take for granted that there is one button on the wall that makes it all turn off? Is it simply that having colo in the middle of the city is so convenient that it overrides the increased cost and the reduced redundancy that are necessitated by that location? You forgot the default Single Point of Failure in anything.. HUMANS. Tuc/TBOH
Re: San Francisco Power Outage
[EMAIL PROTECTED] (Jeff Aitken) writes: ..., we had a failure at another datacenter that uses Piller units, which operate on the same basic principle as the Hitec ones. ... i guess i never understood why anyone would install a piller that far from the equator. (it spins like a top, on a vertical axis, and the angular momentum is really quite gigantic for its size -- it's heavy and it spins really really fast -- and i remember asking a piller tech why his machine wasn't tipped slightly southward to account for Coriolis, and he said i was confused. probably i am.) but for north america, whenever i had a choice, i chose hitec. (which spins with an axis parallel to gravity.) -- Paul Vixie
Re: An Internet IPv6 Transition Plan
On 25-jul-2007, at 6:30, Stephen Wilcox wrote: I think the combined effect of these things means - we will not be running into a wall at any time - availability of IPs will slowly decrease over time (as cost slowly increases) I have to disagree here. 10% of the requests are for 90% of the 170 - 200 million IPv4 addresses given out per year. These are going to large broadband ISPs in blocks of a quarter million or (much) larger, upto /8. At some point, the RIRs will be out of large enough blocks to satisfy these requests. Nothing to be done about that. The decrease over time / address market stuff only applies to the 90% of requests for very smal blocks that together only use 17 - 20 million addresses per year. Those can be satisfied from reclaimed address space for years to come.
Re: An Internet IPv6 Transition Plan
I believe that we'll see extensive use of NAT for client-only services (just look at many broadband residential services today), but that won't help business customers who want a block for the DMZ servers. think a few million /27s or /29s with publicly accessible services on one of those addresses. randy
Re: An Internet IPv6 Transition Plan
On 24-jul-2007, at 0:41, Durand, Alain wrote: 1) What is the IPv6 'service'? For example, is it reasonable to define a 'basic' level service as web+mail and an 'extended' service as everything else? Random ideas include for example offering a lower cost 'basic' service with v6 that would be 'proxied' to the rest of the v4 Internet I would say that IPv6 service is the ability to send packets to and receive packets from other systems also using the IPv6 service by being connected to the global IPv6 cloud. This means that if there is filtering, this must be under the control of the user. Interconnection with IPv4 is a separate problem, and I'm certainly in favor of proxying to achieve that for users who don't need to run more complex protocols over IPv4: http://www.ietf.org/internet-drafts/draft-van-beijnum-v6ops-connect- method-00.txt Hopefully, this will make it possible to start removing IPv4 from select parts of the network: http://arstechnica.com/news.ars/post/20070704-the-declaration-of-ipv6- independence.html 2) What is the connectivity model in IPv6 for the residential customer? 1 address versus prefix delegation? Prefix of course. what prefix size? /48 is a nice round number, but even /64 will do the job for residential users. is this prefix 'stable' or 'variable' over time? (ie renumbering is expected) (note: the answer to this question has huge implications) As a residential ISP, you have to build the network, so you tell us. As long as the prefixes don't change too often and everything is done carefully, user impact is negligible. What types of devices are connected? PCs or appliances or sensors? Nobody knows, and why should you care? What is the management model in the home? Mostly: N/A. Are there 'servers' (ie things that answers connections from the outside) in the home? Of course. Is there any kind of DNS delegation happening to the home? You can't just give every address a name like with IPv4 and you don't really know what addresses customers are going to use. Solution: dynamic DNS. Problem: the authentication. Solution: set up a zone per customer that can be modified with DDNS from the addresses given out to the customer. Bonus: web interface for removing old crap. 3) What is the security model of all this? Javascript is enabled, so: broken. I just listened today half mistified to a presentation at IETF that was saying that the 'recommended' deployment model in the home is to put a NAT-like stateful firewall in the home gateway... This would mean that IPv6 would have to inherit all the NAT- traversal technologies from IPv4 to work... Is this really what we want? No, but how do we avoid it? Vendors need to build good stuff and let the customer make their own decisions in the end, when security stuff gets in the way it WILL be disabled or worked around. 4) What about the 'legacy' devices that cannot upgrade to IPv6? What kind of service is expected for those? Does defining an 80% type solution as in 1) take care of them? Start charging more for IPv4 / less for IPv6, smart users will have a garage sale and buy new stuff, conservative ones do nothing and pay you the extra couple of bucks until 2023.
Re: Why do we use facilities with EPO's?
Seems like the EPO should be a logical AND with the fire alarm system - it only works AFTER you have an existing fire alarm in the building. No, no. If the fire alarm system fails, the fire responders need to be able to hit the EPO and be sure that it works anyways. It has to be an absolute - firefighters have to know that the thing they hit was the only, and right, thing, and that they aren't going to die because they sprayed water on an energized but on fire electrical system backed by a 120 KVA UPS or some such. Also, one should not wait for fire alarms to go off to de-energize a room in the clear presence of an electrical fire or major short. Preventing the fire is better than putting it out. Telco central offices are somewhat of an exception in many ways, but just about anyone else should have a real live EPO. -george william herbert [EMAIL PROTECTED]
Re: Why do we use facilities with EPO's?
Speaking on Deep Background, the Press Secretary whispered: Many years ago, the safety department for my employer made a big stink over the fact that the EPO hadn't been tested in a couple of years. We scheduled an outage window, shut everything down. The facilities guy pressed the magic big RED button and NOTHING! Tracing the problem back, there was a blown fuse in the EPO circuit because a wire had shorted. A real safe design! I've never designed or looked into a EPO installation; but I'm astonished such does not use a Normally-Closed pushbutton in a fail-to-off circuit. Similarly... If you have electric locks on your exit doors; every installation I have seen has a couple of such aspects: a) You must have an exit override. If an electric strike, an interior knob is good. If a [Locknetics-style] mag-lock, you need an exit button. That button SHALL be a NC pushbutton in series with the magnet. [In other words... No, you can't have the pushbutton connected back to some controller box on the 3rd floor where it generates an interupt that will drop the lock power... or it's supposed to...] b) When the building fire drop is pulled, you SHALL drop the lock power to the mag locks. And while local fire codes vary widely; given those were in the rules for a USG SCIF I worked in; I somehow doubt you'll be able to get more lenient treatment based on the import of the data center's operation. -- A host is a host from coast to [EMAIL PROTECTED] no one will talk to a host that's close[v].(301) 56-LINUX Unless the host (that isn't close).pob 1433 is busy, hung or dead20915-1433
Re: Why do we use facilities with EPO's?
If you don't have water-based fire suppression, have normally unoccupied spaces, and are continuously manned, it's sometimes possible to pass on having an EPO. YMMV by inspector. That is indeed true, as we were able to have ours disconnected, and were able to expand our facility without adding EPOs in the new datacenter rooms. We were told that since we met a specific list of criteria, which included things like FM200/Ecaro25 fire suppression, solid (not raised) floors, electrical wiring done some particular way, and large signage indicating where the safety hazards lie for our friends in the Tukwila Seattle Fire Departments... we got a pass on the Big Red Button. I imagine those criteria change from place to place, and time to time... perhaps even inspector to inspector. I sleep better at night. Well... A little bit better. --chuck
Re: History of the EPO (Emergency Power Off)
From: Sean Donelan [EMAIL PROTECTED] Subject: History of the EPO (Emergency Power Off) The interesting thing about the EPO and data centers is it wasn't orginally for life-safety, but came out of a recommendation by IBM to the NFPA for property protection. Fwiw, the EPO on IBM's mainframes back in those days, had to be -pulled- and had a mechanical 'latch' that kept it from being pushed back in. Took both hands to reset it. --Michael
History of the EPO (Emergency Power Off)
The interesting thing about the EPO and data centers is it wasn't orginally for life-safety, but came out of a recommendation by IBM to the NFPA for property protection. But like many things, the original reasoning been lost to history, and the codes grew in different ways. http://www.datacenterknowledge.com/archives/2007/May/07/averting_disaster_with_the_epo_button.html The history of the emergency power off switch dates back to 1959, when a fire in the Air Force's statistical division in the Pentagon caused $6.9 million in property damage and destroyed three IBM mainframe computers. Nothing gets the government.s attention like something that happens to government, said Sawyer. The National Fire Protection Agency (NFPA) was tasked to develop rules to address fire risks in IT environments. Sometimes you need to revisit the rules. For example, for folks thought having automatic water sprinklers in data centers was a bad thing. Slowly folks have started to rethink it, and now automatic sprinklers are found in more data centers. I don't have hard data, but my experience is there have been fewer outages from accidental sprinkler discharges than from accidental EPO activations.
Re: Why do we use facilities with EPO's?
Leo Bicknell wrote: I was complaining to some of the power designers during the building of a major facility that the EPO button represented a single point of failure, and effectively made all of the redundancy built into the power system useless. After all, what's the point of having two (or more) of anything, if there's one button somewhere that turns it all off? Seems like the EPO should be a logical AND with the fire alarm system - it only works AFTER you have an existing fire alarm in the building. -- Mark Radabaugh Amplex 419.837.5015 x21 [EMAIL PROTECTED]
Re: San Francisco Power Outage
Michael Dillon writes: And the stories that the power guy I'm working with tells about foreign facilities, particularly in middle east war zones, are really scary... We fundamentally do not have the facilities problem completely nailed down to the point that things will never drop. Level 4 datacenters can, and will, fail. Nothing you can do including just doing 48V DC for everything are truly foolproof solutions. A single level 4 datacenter is a Single Point of Failure! Two of those middle-eastern style facilities is... ? Has anyone actually kept track of all these data center failures over the years and done some statistical analysis on it? Maybe two half-baked data centers is better than one over the long run? Remember that one 10-12 years ago in (Palo Alto, Mountainview?) where a lady in a car caused a backhoe driver to move out of the way which resulted in him cutting a gas line which resulted in the fire department evacuating the data center, cutting off electricity in the area, and forbidding the diesel generators to be switched on? Santa Clara. I was working right outside the evacuation radius. Which exchange point was in the building? PB-NAP? CIX? I remember we had a net-dark event associated, but not which one. It was a bad day... The lesson, as you point out, is that geographical redundancy is sometimes necessary. This is as true for providers as for datacenter end-users... -george william herbert [EMAIL PROTECTED]
Re: History of the EPO (Emergency Power Off)
At 08:10 PM 7/25/2007, Sean Donelan wrote: Sometimes you need to revisit the rules. For example, for folks thought having automatic water sprinklers in data centers was a bad thing. Slowly folks have started to rethink it, and now automatic sprinklers are found in more data centers. I don't have hard data, but my experience is there have been fewer outages from accidental sprinkler discharges than from accidental EPO activations. There was an interesting study conducted by the US Air Force about fires and other failure modes in computing facilities protected with Halon/FM200/FE227 vs. dry pipe preaction. I know I saved the PDF, but I can't seem to find it at the moment. If my memory is correct, it boiled down to the fact that there had only been two fire incidents at all US Air Force installations and both were due to (surprise, surprise) human factors. One was a stray incendiary munition which breached the datacenter and other was due to a Jet A fuel spill and fire - which is odd because it is hard to ignite kero, diesel, jet A without atomization. The point of the study was that there was zero damage over a 30 year period from water based fire protection systems and I suspect it was pretty handy to have sprinklers when both datacenter fires happened. The munition breach of the physical structure would have rendered any gas based fire suppression system ineffective. In theory, I'm not a big fan of EPOs due to the Is this the button to exit/open the door? problem. One of our redundant 150KVA UPS units caught fire a couple years ago, the input breaker became the EPO since the on-board front panel EPO was completely ineffective (and it still would have been ineffective had it been connected to an external EPO button.) That incident prompted a design change in all of our new datacenter power systems since and all existing systems were also updated. Now all UPS units have separate input and bypass breakers and feeds. Previously we used a single feed, but you can't isolate a burning UPS without dropping your attached load when they share a single breaker and are tied together inside the unit where the fire is happening. Having discrete A B power systems is also a very good thing! Many years ago when we were much, much smaller, the EPO was wired to a special EPO circuit breaker on the main panel which fed the subpanel for the datacenter room. A short on that breaker was like pressing the test switch on a GFCI breaker. Do most people who do have functional (as opposed to decorative) EPO buttons have them connected to the building/suite mains disconnect? or to the output of your UPS units? to a special EPO panel which trips the EPO cutoffs on other units? -Robert Tellurian Networks - Global Hosting Solutions Since 1995 http://www.tellurian.com | 888-TELLURIAN | 973-300-9211 Well done is better than well said. - Benjamin Franklin
Re: ASN Name of the week
On Wed, 25 Jul 2007, Carlos Friacas wrote: We'll probably run out of v4 addresses sooner than 2 byte ASN, No. however, globally it seems more pieces of the puzzle are in place for the latter revolution. Depends on what you define as in place but I would disagree that world is ready to move to ipv6 right away, where as moving to 32-bit ASNs is relatively easy even if some are not ready. -- William Leibzon Elan Networks [EMAIL PROTECTED]
Re: ASN Name of the week
Hi Carlos, We'll probably run out of v4 addresses sooner than 2 byte ASN, however, globally it seems more pieces of the puzzle are in place for the latter revolution. What percentage of your core routers can be configured with a four-octet ASN? :) Cheers, Rob
Re: Why do we use facilities with EPO's?
On Jul 25, 2007, at 2:03 PM, Tuc at T-B-O-H.NET wrote: If they can be avoided, why do we put up with them? Do we really want our colo in downtown San Francisco bad enough to take the risk of having a single point of failure? How can we, as engineers, ask questions about how many generators, how much fuel, and yet take for granted that there is one button on the wall that makes it all turn off? Is it simply that having colo in the middle of the city is so convenient that it overrides the increased cost and the reduced redundancy that are necessitated by that location? You forgot the default Single Point of Failure in anything.. HUMANS. The earth is a SPoF. Let's put DCs on the moon. Besides, safety always overrides convenience. And I don't think that is a bad trade off. -- TTFN, patrick
Re: Why do we use facilities with EPO's?
Funny story about that and the EPO we have here... We have chilled water cooling in our server rooms. A couple of years ago we told the facilities guys there was sand in the lines. They didn't believe us. This went back and forth for a few months until the lines finally ground to a halt. They admitted sand was in the lines. The bring out an HVAC guy... he closes the valve, opens the pipe, nothing comes out. He **opens** the valve, nothing comes out. He whacks on the pipes with a wrench, all the sand and lots of water come out very fast. By the time I got down there, the ceiling tiles were drenched and looked more like sponges. Half the room was soaked. That would be a good reason to have an EPO right there. :) j On 7/25/07, Leo Bicknell [EMAIL PROTECTED] wrote: I was complaining to some of the power designers during the building of a major facility that the EPO button represented a single point of failure, and effectively made all of the redundancy built into the power system useless. After all, what's the point of having two (or more) of anything, if there's one button somewhere that turns it all off? What I found interesting is that a single EPO is not a hard and fast rule. They walked me through a twisty maze of the national electric code, the national fire code, and local regulations. Through that journey, they left me with a rather interesting tidbit. The more urban an area the more likely it is to have strict fire codes. Typically these codes require a single EPO for the entire structure, there's no way to compartmentalize to rooms or subsystems. However in more rural areas this is often not so, and they had in fact built data centers to code WITHOUT a single building EPO in several locations. That's to say there was no EPO, but that it may only affect a single room, or even a single device. If they can be avoided, why do we put up with them? Do we really want our colo in downtown San Francisco bad enough to take the risk of having a single point of failure? How can we, as engineers, ask questions about how many generators, how much fuel, and yet take for granted that there is one button on the wall that makes it all turn off? Is it simply that having colo in the middle of the city is so convenient that it overrides the increased cost and the reduced redundancy that are necessitated by that location? -- Leo Bicknell - [EMAIL PROTECTED] - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ Read TMBG List - [EMAIL PROTECTED], www.tmbg.org -- /* [EMAIL PROTECTED] is an alias for all ISC handlers. Please include the list in all replies to keep everyone informed. You may receive more than one response */ Thanks, j
Re: Why do we use facilities with EPO's?
On 7/25/07, Leo Bicknell [EMAIL PROTECTED] wrote: The more urban an area the more likely it is to have strict fire codes. snipped If they can be avoided, why do we put up with them? Do we really want our colo in downtown San Francisco bad enough to take the risk of having a single point of failure? How can we, as engineers, ask questions about how many generators, how much fuel, and yet take for granted that there is one button on the wall that makes it all turn off? Is it simply that having colo in the middle of the city is so convenient that it overrides the increased cost and the reduced redundancy that are necessitated by that location? We put up with EPOs for the same reason we put up with water-based fire suppression systems. Safety. When my firefighter buddy needs to cut through a wall in a data center, she better damn well be sure she can kill power to the entire area before she potentially cuts through 400-500V feeds in the walls. Hence, the EPO. Now, I know you referred to a single EPO for the entire facility, and how that's akin to doing brain surgery with a sledgehammer. Perhaps someday someone will come up with a more surgical method for ensuring power is removed from the areas it needs to, while other unaffected areas can continue functioning. Until that point though, I prefer we err on the side of caution and value someone's life over business continuity. Almost forget. You mentioned more urban areas require the master EPO being easily accessible in the facility. I don't know what gets more urban than the San Fran area. -brandon
Re: Why do we use facilities with EPO's?
At 12:07 PM -0400 7/25/07, Leo Bicknell wrote: The more urban an area the more likely it is to have strict fire codes. Typically these codes require a single EPO for the entire structure, there's no way to compartmentalize to rooms or subsystems. For high-availability sites (Tier III, Tier IV per UpTime Institute), EPO's are one of the most common reasons for outage. I'd highly recommend APC's paper http://www.apcmedia.com/salestools/ASTE-5T3TTT_R2_EN.pdf on the topic. Short-version is that its a safety issue for room occupants and responders. More mature codes tend to require such, particularly in the presence of UPS gear which can be energized unbeknownst to fire fighting personnel. If you don't have water-based fire suppression, have normally unoccupied spaces, and are continuously manned, it's sometimes possible to pass on having an EPO. YMMV by inspector. /John
RE: Why do we use facilities with EPO's?
In fact, an EPO system is a single point of failure... And, whether or not you need an EPO in your center is wholly up to you, and how you design your center. As mentioned at a recent seminar I went to: If you do not need to install non-plenum rated cable below a floor, and you require boxes under the floor to be secured, and you do not state NFPA 75 as your standard, then you do not need an EPO as defined by NEC 645. Only if you want exceptions granted in 645 (Information Technology Equipment), should you have to install an EPO. EPO = SPOF = bad. We all know this. If they can be avoided, why do we put up with them? Do we really want our colo in downtown San Francisco bad enough to take the risk of having a single point of failure? How can we, as engineers, ask questions about how many generators, how much fuel, and yet take for granted that there is one button on the wall that makes it all turn off? Is it simply that having colo in the middle of the city is so convenient that it overrides the increased cost and the reduced redundancy that are necessitated by that location? You forgot the default Single Point of Failure in anything.. HUMANS. Tuc/TBOH
