We wanted to provide you with some additional detail regarding our recent outage. On July 2nd, between 6:20 AM PT and 12:30 PM PT, all applications experienced increased error rate and latency with Datastore and memcache operations, as well as some serving errors. Datastore access and serving were fully restored as of 12:25 PM PT.
Problem There was a serious issue in one of App Engine's datacenters with GFS, Google's low level storage system. GFS underlies Bigtable, which in turn underlies App Engine's Datastore. GFS also provides storage for our application serving infrastructure, so GFS unavailability caused problems for Datastore reads and writes, as well as application serving. Resolution Efforts Availability and data integrity are both very important to the App Engine team. Typically, we would have switched to an alternate datacenter immediately. However, due to the specific nature of this problem, switching datacenters immediately meant that the most recent data written by applications would not have been available, leading to consistency problems for many applications. The team decided to try to stabilize GFS first, then switch datacenters. This was accomplished and we avoided any data consistency issues. Prevention The team has been actively working on a solution in the medium-term that would allow us to switchover datacenters immediately without consistency problems. Communication and Status Many users noted that the System Status site was also down. The System Status site is hosted separately from App Engine applications, and is not typically affected by availability problems. However, due to the low level problem with GFS in this case, the System Status site was also affected. The team did post the downtime announcement and updates on the Downtime Notification group, available here: http://groups.google.com/group/google-appengine-downtime-notify The App Engine team is continuing to work to improve the availability and power of App Engine. Thanks for your patience. Chris Beckmann Product Manager, App Engine Team --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---
