I posted an issue for making it easier for developers to do their own testing with disabled capabilities:
http://code.google.com/p/googleappengine/issues/detail?id=1811 Having inconsistent data could lead to problems that lingered for a long time, particularly since memcache will be out of sync, I think, so you made the right call on not rolling over quickly. On Jul 2, 7:47 pm, "Chris Beckmann (App Engine PM)" <beckmann +...@google.com> wrote: > We wanted to provide you with some additional detail regarding our > recent outage. On July 2nd, between 6:20 AM PT and 12:30 PM PT, all > applications experienced increased error rate and latency with > Datastore and memcache operations, as well as some serving errors. > Datastore access and serving were fully restored as of 12:25 PM PT. > > Problem > > There was a serious issue in one of App Engine's datacenters with GFS, > Google's low level storage system. GFS underlies Bigtable, which in > turn underlies App Engine's Datastore. GFS also provides storage for > our application serving infrastructure, so GFS unavailability caused > problems for Datastore reads and writes, as well as application > serving. > > Resolution Efforts > > Availability and data integrity are both very important to the App > Engine team. Typically, we would have switched to an alternate > datacenter immediately. However, due to the specific nature of this > problem, switching datacenters immediately meant that the most recent > data written by applications would not have been available, leading to > consistency problems for many applications. > > The team decided to try to stabilize GFS first, then switch > datacenters. This was accomplished and we avoided any data consistency > issues. > > Prevention > > The team has been actively working on a solution in the medium-term > that would allow us to switchover datacenters immediately without > consistency problems. > > Communication and Status > > Many users noted that the System Status site was also down. The System > Status site is hosted separately from App Engine applications, and is > not typically affected by availability problems. However, due to the > low level problem with GFS in this case, the System Status site was > also affected. The team did post the downtime announcement and updates > on the Downtime Notification group, available > here:http://groups.google.com/group/google-appengine-downtime-notify > > The App Engine team is continuing to work to improve the availability > and power of App Engine. Thanks for your patience. > > Chris Beckmann > Product Manager, App Engine Team --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---
