RE: [google-appengine] Massive EC2 outage

2011-04-25 Thread Brandon Wirtz
-appengine] Massive EC2 outage In the event of a primary data center outage, we would failover to a secondary data center. If you are using High Replication datastore, you should not experience any downtime while this happens. In addition, any data that are successfully committed will have been

Re: [google-appengine] Massive EC2 outage

2011-04-25 Thread Ikai Lan (Google)
In the event of a primary data center outage, we would failover to a secondary data center. If you are using High Replication datastore, you should not experience any downtime while this happens. In addition, any data that are successfully committed will have been guaranteed to have been written.

Re: [google-appengine] Massive EC2 outage

2011-04-25 Thread kowsik
It's less about the data-center going down, but more about App design and distributing it across regions. All the apps that relied on their PaaS vendors that only provisioned in the us-east-1 region are all hurting right now (including us). If only PaaS offerings enabled regional affinity along wit

Re: [google-appengine] Massive EC2 outage

2011-04-21 Thread Robert Kluin
Hopefully they can get everything sorted out soon. Depends on your app, but having a bit of downtime sprinkled across the year in small doses is probably easier to handle than a massive outage that shuts down a site for a day. Any chance a Googler could comment about how well distributed and inde

Re: [google-appengine] Massive EC2 outage

2011-04-21 Thread saidimu apale
FWIW, here's DotCloud's take on the outage which is still affecting them: http://blog.dotcloud.com/working-around-the-ec2-outage saidimu On Thu, Apr 21, 2011 at 6:12 PM, Jeff Schnitzer wrote: > On Thu, Apr 21, 2011 at 1:58 PM, saidimu apale wrote: > > > > It sure helps that you have a batph

Re: [google-appengine] Massive EC2 outage

2011-04-21 Thread Jeff Schnitzer
On Thu, Apr 21, 2011 at 1:58 PM, saidimu apale wrote: > > It sure helps that you have a batphone to the GAE datastore team. Or is that > a different Jeff Schnitzer? Don't interpret that too literally. A couple of the Google developers have been receptive to some api changes that would make Objec

Re: [google-appengine] Massive EC2 outage

2011-04-21 Thread saidimu apale
> I've been there and I'm happy to leave this job to the professionals. It sure helps that you have a batphone to the GAE datastore team. Or is that a different Jeff Schnitzer? saidimu On Thu, Apr 21, 2011 at 1:14 PM, Jeff Schnitzer wrote: > Easy to say, not so easy to do :-) > > All that fra

Re: [google-appengine] Massive EC2 outage

2011-04-21 Thread Derrick Schneider
Not easy, sure, though EC2 has plenty of support for different availability zones (as does RDS) so it's not impossible (though you pay normal bandwidth charges between availability zones, as opposed to getting it for free). So I guess it boils down to figuring out how much outages cost you. In the

Re: [google-appengine] Massive EC2 outage

2011-04-21 Thread Jeff Schnitzer
Easy to say, not so easy to do :-) All that frantic running-around-in-panic that GAE engineers and ops people do when something goes wonky inside the datastore? That would be you right now, trying to figure out how to failover your database (and any other persistent data) to a different datacente

Re: [google-appengine] Massive EC2 outage

2011-04-21 Thread Derrick Schneider
While it's certainly unfortunate for companies that have bet on EC2, they'd be smarter to distribute their servers a bit so that a single data center outage does not take out their entire company. On Thu, Apr 21, 2011 at 10:02 AM, Jeff Schnitzer wrote: > I'm not suggesting schadenfreude here, but

[google-appengine] Massive EC2 outage

2011-04-21 Thread Jeff Schnitzer
I'm not suggesting schadenfreude here, but for all those folks doubting the viability of appengine for reliability reasons: http://eu.techcrunch.com/2011/04/21/amazon-ec2-goes-down-taking-with-it-reddit-foursquare-and-quora/ Amazon's North Virginia datacenter tripped and fell over in the early AM