Re: [google-appengine] Some instances running unreleased version of the App Engine SDK?

2015-09-18 Thread Nick
I would imagine anyone doing anything half serious has a series of environments, all my projects have at least a -Dev and -UAT environment. I'd much prefer if those got the newer sdk first. Or just actual environmental support built in - that would solve everyone's problem (and was underway at

Re: [google-appengine] Some instances running unreleased version of the App Engine SDK?

2015-09-18 Thread PK
Jesse, agreed. But at a minimum would like to control when it happens so it is a mutually convenient time. BTW, you guys used to do SDK pre-releases. This might also help to catch some issues. I sed to always download the pre-release and run my tests on it. > On Sep 18, 2015, at 1:15 PM, Jesse

Re: [google-appengine] Some instances running unreleased version of the App Engine SDK?

2015-09-18 Thread Jesse Scherer (Google Cloud Support)
Julian there should be somebody from the engineering team reaching out to you soon -- if not already -- to understand what happened in your case. As I mentioned in PK's issue we're working on ways to handle canaries better. BTW the most obvious approach, a flag for "never run my app on a canary

Re: [google-appengine] Some instances running unreleased version of the App Engine SDK?

2015-09-17 Thread Julian Bunn
Thanks, PK - I will take a look at the issue you filed and comment accordingly. We suffered pretty major disruption to our application, and lost many hours of data, unfortunately. The symptoms were extra null fields in some Datastore objects, which caused storage of new uploaded data from clien

Re: [google-appengine] Some instances running unreleased version of the App Engine SDK?

2015-09-17 Thread PK
Hi Julian, Google routinely does this when they release a new runtime that they believe it is backwards incompatible. It mostly works OK but I did experience downtime for several hours on July 24th because a new version was rolled out that had an issue. It was a Friday, I was in an intercontine

[google-appengine] Some instances running unreleased version of the App Engine SDK?

2015-09-17 Thread Julian Bunn
We noticed on Sep 15 and 16 (a couple of days ago) that some of our GAE instances were running version 1.9.27 of the SDK, rather than the released version 1.9.26. (We coincidentally had several serious issues with our application.) Could someone please advise on how this can happen, and what we