Re: [osol-discuss] Opensolaris.org is back up
You want enough battery time to start the generators, and fix the generators and pumps that may possibly fail. (And you definately want to make sure you have redundant fuel pumps for each generator.) Also quarterly testing is a must. --brian On 7/27/07, Bill Sommerfeld [EMAIL PROTECTED] wrote: On Tue, 2007-07-24 at 22:20 -0700, Hugh McIntyre wrote: http://www.prnewswire.com/cgi-bin/stories.pl?ACCT=104STORY=/www/story/07-24-2007/0004631109EDATE= In answer to the questions about backup power, obviously they have UPS and generators (as the press release says...). There is more detail available here: http://www.365main.com/status_update.html Looks like they had 8 primary and 2 secondary generators - N+2 redundancy. They lost 5 -- 4 failed to start, and another was overloaded into failure due to the first 4 failures. At first, three primary and one secondary generator failed to come up. It appears there was sufficient capacity in the remaining secondary generator to handle the load of two of the primaries, but not three, so the running secondary generator *also* fell over. As a result, three of their eight rooms lost power for about 45 minutes; a fourth room lost power for half a second due to some sort of PDU glitch. - Bill ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Opensolaris.org is back up
One more thing. Geographical redundancy is a good thing. -brian On 7/27/07, Brian Gupta [EMAIL PROTECTED] wrote: You want enough battery time to start the generators, and fix the generators and pumps that may possibly fail. (And you definately want to make sure you have redundant fuel pumps for each generator.) Also quarterly testing is a must. --brian On 7/27/07, Bill Sommerfeld [EMAIL PROTECTED] wrote: On Tue, 2007-07-24 at 22:20 -0700, Hugh McIntyre wrote: http://www.prnewswire.com/cgi-bin/stories.pl?ACCT=104STORY=/www/story/07-24-2007/0004631109EDATE= In answer to the questions about backup power, obviously they have UPS and generators (as the press release says...). There is more detail available here: http://www.365main.com/status_update.html Looks like they had 8 primary and 2 secondary generators - N+2 redundancy. They lost 5 -- 4 failed to start, and another was overloaded into failure due to the first 4 failures. At first, three primary and one secondary generator failed to come up. It appears there was sufficient capacity in the remaining secondary generator to handle the load of two of the primaries, but not three, so the running secondary generator *also* fell over. As a result, three of their eight rooms lost power for about 45 minutes; a fourth room lost power for half a second due to some sort of PDU glitch. - Bill ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Opensolaris.org is back up
On Tue, 2007-07-24 at 22:20 -0700, Hugh McIntyre wrote: http://www.prnewswire.com/cgi-bin/stories.pl?ACCT=104STORY=/www/story/07-24-2007/0004631109EDATE= In answer to the questions about backup power, obviously they have UPS and generators (as the press release says...). There is more detail available here: http://www.365main.com/status_update.html Looks like they had 8 primary and 2 secondary generators - N+2 redundancy. They lost 5 -- 4 failed to start, and another was overloaded into failure due to the first 4 failures. At first, three primary and one secondary generator failed to come up. It appears there was sufficient capacity in the remaining secondary generator to handle the load of two of the primaries, but not three, so the running secondary generator *also* fell over. As a result, three of their eight rooms lost power for about 45 minutes; a fourth room lost power for half a second due to some sort of PDU glitch. - Bill ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Opensolaris.org is back up
Dennis Clarke wrote: Nothing can replace batteries. The more the better. Most power solutions should grant at *least* 30 minutes on battery and if not then I would not host my gear there. I just finished installing a 45 minute battery runtime solution with 16,000 VA of power. It is not that costly really to stick in an APC Symmetra SY16K unit when one looks at the cost of downtime. Most datacenters consume a *lot* of power. Just how many lead-acid batteries can you fit in one building? (There are probably safety and weight issues, plus density problems if every other rack in your datacenter is a UPS). And in any case, an outage can just as easily last an hour or more as it can last a few minutes. So the usual plan is that the UPS only needs to last long enough for the generator to kick in, e.g. 5-10 minutes. I've also seen people use flywheels for short term power, but again there's only so large a flywheel the authorities will let you install. Hugh. PS: at an Indian office of the company I work for, we recently managed to have a backup generator last for 2-3 days via continuous refueling because of the failure and wait-for-replacement of a step down transformer. I doubt you could have had enough batteries for this... ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Opensolaris.org is back up
Dennis Clarke wrote: Nothing can replace batteries. The more the better. Most power solutions should grant at *least* 30 minutes on battery and if not then I would not host my gear there. I just finished installing a 45 minute battery runtime solution with 16,000 VA of power. It is not that costly really to stick in an APC Symmetra SY16K unit when one looks at the cost of downtime. Most datacenters consume a *lot* of power. Just how many lead-acid batteries can you fit in one building? (There are probably safety and weight issues, plus density problems if every other rack in your datacenter is a UPS). I wasn't clear. The batteries just give you time to start the generator. Up here in Canada the generator will be frozen and, possibly, the diesel fuel will be also. Provided you have a Jeep with winter dueller tires on it and you can get to the server room in 40 minutes then all you need is another ten or 15 minutes to drag the jerry cans to the generator from the fuel shed which ( hopefully ) is not iced over with the locks frozen. You see .. it can be tricky. But cooling is real cheap. Dennis ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Opensolaris.org is back up
Hugh McIntyre wrote: Dennis Clarke wrote: Nothing can replace batteries. The more the better. Most power solutions should grant at *least* 30 minutes on battery and if not then I would not host my gear there. I just finished installing a 45 minute battery runtime solution with 16,000 VA of power. It is not that costly really to stick in an APC Symmetra SY16K unit when one looks at the cost of downtime. Most datacenters consume a *lot* of power. Just how many lead-acid batteries can you fit in one building? (There are probably safety and weight issues, plus density problems if every other rack in your datacenter is a UPS). The best I've seen was the central telecommunications hub of a country I shall not mention which had 48 hours of battery backup. That was a lot of batteries and yes, it was on the ground floor! Ian ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Opensolaris.org is back up
Thanks, Tirthankar Stephen Lau wrote: Dennis Clarke wrote: There was a major power outage in SF that affected our servers. three little letters : U P S better yet : A P C get some :-) The whole reason we host in a datacentre is that the datacentre is supposed to take care of things like UPSs and generators. Clearly, something went wrong at the datacentre level and we will follow up with them to make sure things are rectified for future outages. They should start using the Solaris Cluster geo edition http://www.sun.com/software/solaris/cluster/index.xml cheers, steve ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Opensolaris.org is back up
There was a major power outage in SF that affected our servers. three little letters : U P S better yet : A P C get some :-) Dennis ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Opensolaris.org is back up
Wow, I was under the impression it only affected residential areas... This message posted from opensolaris.org ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Opensolaris.org is back up
Wow, I was under the impression it only affected residential areas... I'm guessing netflix, craigslist, typepad and livejournal are not among your favorite sites :) -Artem ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Opensolaris.org is back up
Derek Cicero wrote: There was a major power outage in SF that affected our servers. Discussion search does not seem to be working. For example: http://www.opensolaris.org/jive/search.jspa?objID=f80q=ZFS -Manoj ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Opensolaris.org is back up
Dennis Clarke wrote: There was a major power outage in SF that affected our servers. three little letters : U P S better yet : A P C get some :-) It should have been part of their B C P ;-) ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Opensolaris.org is back up
Wow, I was under the impression it only affected residential areas... I'm guessing netflix, craigslist, typepad and livejournal are not among your favorite sites :) geez .. I think I looked at one of them once .. maybe. as for residential areas, I have 6000 VA worth of UPS here at my desk ( at home ) and even my X-Box has a UPS. Never lose a game to power outage! so why do we not have 10,000 VA worth of APC gear on the OpenSolaris.org servers? geez .. a few batteries are no big deal. Dennis ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Opensolaris.org is back up
geez .. I think I looked at one of them once .. maybe. as for residential areas, I have 6000 VA worth of UPS here at my desk ( at home ) and even my X-Box has a UPS. Never lose a game to power outage! The Warlock: Why did you bring a cop to my command center? John McClane: It's a basement! The Warlock: Who is this man? ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Opensolaris.org is back up
geez .. I think I looked at one of them once .. maybe. as for residential areas, I have 6000 VA worth of UPS here at my desk ( at home ) and even my X-Box has a UPS. Never lose a game to power outage! The Warlock: Why did you bring a cop to my command center? John McClane: It's a basement! The Warlock: Who is this man? I am not *that* much of a geek .. but .. when the power goes out I still have lights, computers, a network switch and a television and the XBox and .. oh no ... maybe I am ... but its not a bloody command center it is just the basement. Really .. its not big deal. No no .. really. dc ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Opensolaris.org is back up
Dennis Clarke wrote: There was a major power outage in SF that affected our servers. three little letters : U P S better yet : A P C get some :-) Heh, sometimes those three letters can themselves give up on your face: http://blogs.sun.com/moinakg/entry/the_belenix_0_6_ride Regards, Moinak. Dennis ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Opensolaris.org is back up
Dennis Clarke wrote: There was a major power outage in SF that affected our servers. three little letters : U P S better yet : A P C get some :-) The whole reason we host in a datacentre is that the datacentre is supposed to take care of things like UPSs and generators. Clearly, something went wrong at the datacentre level and we will follow up with them to make sure things are rectified for future outages. cheers, steve -- stephen lau // [EMAIL PROTECTED] | 650.786.0845 | http://whacked.net opensolaris // solaris kernel development ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Opensolaris.org is back up
Dennis Clarke wrote: There was a major power outage in SF that affected our servers. three little letters : U P S better yet : A P C get some :-) Heh, sometimes those three letters can themselves give up on your face: http://blogs.sun.com/moinakg/entry/the_belenix_0_6_ride on point 25 there .. I agree that APC service is excellent. I have five units here that are 1200 VA each and then a 1500VA unit and a few small guys kicking around. That keeps five or six computers going as well as some other devices. The real trick is to do an orderly shut down because most of these newer guys want USB connections and you just can't plug that into the back of some of my boxes here. the ultimate is the 48V big monster batteries I have in my server room. Two man lift only at about 90Kg each I think. Feels like a ton to lift. But the run time is just amazing at well over 600 minutes. oh well ... I see you have your fair share of APC events in your life also. dc ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Opensolaris.org is back up
Ché Kristo wrote: Wow, I was under the impression it only affected residential areas... More details at: http://www.theregister.co.uk/2007/07/24/web_site_outages/ Particularly sad is the comment in the article that: Redenvelope continued to be unavailable, the same day PR monkeys issued a press release celebrating the site's two years of 100-percent uptime at 365 Main's San Francisco facility. Oops. Oops indeed. For some reason the press release now produces a 404 error ;). Although it's available elsewhere at: http://www.prnewswire.com/cgi-bin/stories.pl?ACCT=104STORY=/www/story/07-24-2007/0004631109EDATE= In answer to the questions about backup power, obviously they have UPS and generators (as the press release says...). But such backup systems only get tested rarely, and have an unfortunate tendency to not always work just when they are needed. No data center can run on UPS for more than a few minutes even with the largest UPS, so they all need to have generators. Hugh. ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Opensolaris.org is back up
Ché Kristo wrote: Wow, I was under the impression it only affected residential areas... More details at: http://www.theregister.co.uk/2007/07/24/web_site_outages/ Particularly sad is the comment in the article that: Redenvelope continued to be unavailable, the same day PR monkeys issued a press release celebrating the site's two years of 100-percent uptime at 365 Main's San Francisco facility. Oops. Oops indeed. For some reason the press release now produces a 404 error ;). Although it's available elsewhere at: http://www.prnewswire.com/cgi-bin/stories.pl?ACCT=104STORY=/www/story/07-24-2007/0004631109EDATE= In answer to the questions about backup power, obviously they have UPS and generators (as the press release says...). But such backup systems only get tested rarely, and have an unfortunate tendency to not always work just when they are needed. No data center can run on UPS for more than a few minutes even with the largest UPS, so they all need to have generators. Nothing can replace batteries. The more the better. Most power solutions should grant at *least* 30 minutes on battery and if not then I would not host my gear there. I just finished installing a 45 minute battery runtime solution with 16,000 VA of power. It is not that costly really to stick in an APC Symmetra SY16K unit when one looks at the cost of downtime. Dennis ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
Re: [osol-discuss] Opensolaris.org is back up
correct! This message posted from opensolaris.org ___ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
