Thanks, PK - I will take a look at the issue you filed and comment accordingly.
We suffered pretty major disruption to our application, and lost many hours of data, unfortunately. The symptoms were extra null fields in some Datastore objects, which caused storage of new uploaded data from clients to fail, and a greatly increased Error rate from our app. Julian On Thursday, September 17, 2015 at 2:22:08 PM UTC-7, PK wrote: > > Hi Julian, > > Google routinely does this when they release a new runtime that they > believe it is backwards incompatible. It mostly works OK but I did > experience downtime for several hours on July 24th because a new version > was rolled out that had an issue. It was a Friday, I was in an > intercontinental flight without internet when the roll out happened and it > was a small disaster. Google did take the issue seriously when I finally > brought it to their attention but it took many hours to get back to stable > ground. > > I think Google can do some things to improve this. First, it needs to pass > some control of when we upgrade the runtime to developers hands so the > world does not change suddenly and unexpectedly under our feet. > Furthermore, as they roll out new versions they should *automatically* > revert applications back to the previous version if the error rate was > almost zero before and spikes as soon as instances in the new version come > onboard. > > I have filed this issue > <https://code.google.com/p/googleappengine/issues/detail?id=12355> in the > public tracker. I highly encourage others to star it and add your ideas > before you find yourselves in my or Julian’s shoes. > > Best > > > On Sep 17, 2015, at 12:34 PM, Julian Bunn <jjb...@gmail.com <javascript:>> > wrote: > > We noticed on Sep 15 and 16 (a couple of days ago) that some of our GAE > instances were running version 1.9.27 of the SDK, rather than the released > version 1.9.26. > > (We coincidentally had several serious issues with our application.) > > Could someone please advise on how this can happen, and what we can do to > prevent it happening in the future (i.e. force instances to only run the > latest production release of the SDK)? Is there an application setting > somewhere? > > I apologize if this is the wrong Group for such questions. > > Many thanks! > Julian > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to google-appengi...@googlegroups.com <javascript:>. > To post to this group, send email to google-a...@googlegroups.com > <javascript:>. > Visit this group at http://groups.google.com/group/google-appengine. > To view this discussion on the web visit > https://groups.google.com/d/msgid/google-appengine/4ac6bcf5-9a32-40fe-ac5a-0886e4eceeb9%40googlegroups.com > > <https://groups.google.com/d/msgid/google-appengine/4ac6bcf5-9a32-40fe-ac5a-0886e4eceeb9%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > > > > PK > p...@gae123.com <javascript:> > > > > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/47f9d861-41de-487d-bc29-355dfd70c253%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
