[google-appengine] Some instances running unreleased version of the App Engine SDK?

2015-09-17 Thread Julian Bunn 
We noticed on Sep 15 and 16 (a couple of days ago) that some of our GAE 
instances were running version 1.9.27 of the SDK, rather than the released 
version 1.9.26.

(We coincidentally had several serious issues with our application.)

Could someone please advise on how this can happen, and what we can do to 
prevent it happening in the future (i.e. force instances to only run the 
latest production release of the SDK)? Is there an application setting 
somewhere?

I apologize if this is the wrong Group for such questions.

Many thanks!
Julian

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/4ac6bcf5-9a32-40fe-ac5a-0886e4eceeb9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Some instances running unreleased version of the App Engine SDK?

2015-09-17 Thread PK 
Hi Julian,

Google routinely does this when they release a new runtime that they believe it 
is backwards incompatible. It mostly works OK but I did experience downtime for 
several hours on July 24th because a new version was rolled out that had an 
issue. It was a Friday, I was in an intercontinental flight without internet 
when the roll out happened and it was a small disaster. Google did take the 
issue seriously when I finally brought it to their attention but it took many 
hours to get back to stable ground. 

I think Google can do some things to improve this. First, it needs to pass some 
control of when we upgrade the runtime to developers hands so the world does 
not change suddenly and unexpectedly under our feet. Furthermore, as they roll 
out new versions they should automatically revert applications back to the 
previous version if the error rate was almost zero before and spikes as soon as 
instances in the new version come onboard.

I have filed this issue 
 in the 
public tracker. I highly encourage others to star it and add your ideas before 
you find yourselves in my or Julian’s shoes.

Best 


> On Sep 17, 2015, at 12:34 PM, Julian Bunn  wrote:
> 
> We noticed on Sep 15 and 16 (a couple of days ago) that some of our GAE 
> instances were running version 1.9.27 of the SDK, rather than the released 
> version 1.9.26.
> 
> (We coincidentally had several serious issues with our application.)
> 
> Could someone please advise on how this can happen, and what we can do to 
> prevent it happening in the future (i.e. force instances to only run the 
> latest production release of the SDK)? Is there an application setting 
> somewhere?
> 
> I apologize if this is the wrong Group for such questions.
> 
> Many thanks!
> Julian
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> .
> To post to this group, send email to google-appengine@googlegroups.com 
> .
> Visit this group at http://groups.google.com/group/google-appengine 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/4ac6bcf5-9a32-40fe-ac5a-0886e4eceeb9%40googlegroups.com
>  
> .
> For more options, visit https://groups.google.com/d/optout 
> .


PK
p...@gae123.com




-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/9944A742-761C-4567-8AB6-E0862DB8E7CD%40gae123.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Some instances running unreleased version of the App Engine SDK?

2015-09-17 Thread Julian Bunn 
Thanks, PK - I will take a look at the issue you filed and comment 
accordingly.

We suffered pretty major disruption to our application, and lost many hours 
of data, unfortunately. The symptoms were extra null fields in some 
Datastore objects, which caused storage of new uploaded data from clients 
to fail, and a greatly increased Error rate from our app. 

Julian



On Thursday, September 17, 2015 at 2:22:08 PM UTC-7, PK wrote:
>
> Hi Julian,
>
> Google routinely does this when they release a new runtime that they 
> believe it is backwards incompatible. It mostly works OK but I did 
> experience downtime for several hours on July 24th because a new version 
> was rolled out that had an issue. It was a Friday, I was in an 
> intercontinental flight without internet when the roll out happened and it 
> was a small disaster. Google did take the issue seriously when I finally 
> brought it to their attention but it took many hours to get back to stable 
> ground. 
>
> I think Google can do some things to improve this. First, it needs to pass 
> some control of when we upgrade the runtime to developers hands so the 
> world does not change suddenly and unexpectedly under our feet. 
> Furthermore, as they roll out new versions they should *automatically* 
> revert applications back to the previous version if the error rate was 
> almost zero before and spikes as soon as instances in the new version come 
> onboard.
>
> I have filed this issue 
>  in the 
> public tracker. I highly encourage others to star it and add your ideas 
> before you find yourselves in my or Julian’s shoes.
>
> Best 
>
>
> On Sep 17, 2015, at 12:34 PM, Julian Bunn > 
> wrote:
>
> We noticed on Sep 15 and 16 (a couple of days ago) that some of our GAE 
> instances were running version 1.9.27 of the SDK, rather than the released 
> version 1.9.26.
>
> (We coincidentally had several serious issues with our application.)
>
> Could someone please advise on how this can happen, and what we can do to 
> prevent it happening in the future (i.e. force instances to only run the 
> latest production release of the SDK)? Is there an application setting 
> somewhere?
>
> I apologize if this is the wrong Group for such questions.
>
> Many thanks!
> Julian
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengi...@googlegroups.com .
> To post to this group, send email to google-a...@googlegroups.com 
> .
> Visit this group at http://groups.google.com/group/google-appengine.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/4ac6bcf5-9a32-40fe-ac5a-0886e4eceeb9%40googlegroups.com
>  
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>
>
>
> PK
> p...@gae123.com 
>
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/47f9d861-41de-487d-bc29-355dfd70c253%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Some instances running unreleased version of the App Engine SDK?

2015-09-18 Thread Jesse Scherer (Google Cloud Support) 
Julian there should be somebody from the engineering team reaching out to 
you soon -- if not already -- to understand what happened in your case.

As I mentioned in PK's issue we're working on ways to handle canaries 
better. BTW the most obvious approach, a flag for "never run my app on a 
canary app server" is not that great because it would simply delay 
discovery of problems until the appserver version which works for you is no 
longer current.

On Thursday, September 17, 2015 at 6:32:17 PM UTC-4, Julian Bunn wrote:
>
> Thanks, PK - I will take a look at the issue you filed and comment 
> accordingly.
>
> We suffered pretty major disruption to our application, and lost many 
> hours of data, unfortunately. The symptoms were extra null fields in some 
> Datastore objects, which caused storage of new uploaded data from clients 
> to fail, and a greatly increased Error rate from our app. 
>
> Julian
>
>
>
> On Thursday, September 17, 2015 at 2:22:08 PM UTC-7, PK wrote:
>>
>> Hi Julian,
>>
>> Google routinely does this when they release a new runtime that they 
>> believe it is backwards incompatible. It mostly works OK but I did 
>> experience downtime for several hours on July 24th because a new version 
>> was rolled out that had an issue. It was a Friday, I was in an 
>> intercontinental flight without internet when the roll out happened and it 
>> was a small disaster. Google did take the issue seriously when I finally 
>> brought it to their attention but it took many hours to get back to stable 
>> ground. 
>>
>> I think Google can do some things to improve this. First, it needs to 
>> pass some control of when we upgrade the runtime to developers hands so the 
>> world does not change suddenly and unexpectedly under our feet. 
>> Furthermore, as they roll out new versions they should *automatically* 
>> revert applications back to the previous version if the error rate was 
>> almost zero before and spikes as soon as instances in the new version come 
>> onboard.
>>
>> I have filed this issue 
>>  in 
>> the public tracker. I highly encourage others to star it and add your ideas 
>> before you find yourselves in my or Julian’s shoes.
>>
>> Best 
>>
>>
>> On Sep 17, 2015, at 12:34 PM, Julian Bunn  wrote:
>>
>> We noticed on Sep 15 and 16 (a couple of days ago) that some of our GAE 
>> instances were running version 1.9.27 of the SDK, rather than the released 
>> version 1.9.26.
>>
>> (We coincidentally had several serious issues with our application.)
>>
>> Could someone please advise on how this can happen, and what we can do to 
>> prevent it happening in the future (i.e. force instances to only run the 
>> latest production release of the SDK)? Is there an application setting 
>> somewhere?
>>
>> I apologize if this is the wrong Group for such questions.
>>
>> Many thanks!
>> Julian
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Google App Engine" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to google-appengi...@googlegroups.com.
>> To post to this group, send email to google-a...@googlegroups.com.
>> Visit this group at http://groups.google.com/group/google-appengine.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/google-appengine/4ac6bcf5-9a32-40fe-ac5a-0886e4eceeb9%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>> PK
>> p...@gae123.com
>>
>>
>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/9efce746-8638-4201-a600-283eb4e83a0e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Some instances running unreleased version of the App Engine SDK?

2015-09-18 Thread PK 
Jesse, agreed. But at a minimum would like to control when it happens so it is 
a mutually convenient time.

BTW, you guys used to do SDK pre-releases. This might also help to catch some 
issues. I sed to always download the pre-release and run my tests on it.

> On Sep 18, 2015, at 1:15 PM, Jesse Scherer (Google Cloud Support) 
>  wrote:
> 
> BTW the most obvious approach, a flag for "never run my app on a canary app 
> server" is not that great because it would simply delay discovery of problems 
> until the appserver version which works for you is no longer current.


PK
p...@gae123.com




-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/31EF6E6A-B1A5-464F-BA1E-B2230BD59CCA%40gae123.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Some instances running unreleased version of the App Engine SDK?

2015-09-18 Thread Nick 
I would imagine anyone doing anything half serious has a series of 
environments, all my projects have at least a -Dev and -UAT environment. I'd 
much prefer if those got the newer sdk first.

Or just actual environmental support built in - that would solve everyone's 
problem (and was underway at one point, right?)

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/4ef1b563-19ca-4d16-8b1f-c227fcd48130%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.