[google-appengine] Re: Alert triggered for 500 responses, but no 500 responses in logs

[Alan deLespinasse]

Hmm, ok, with help from this post 
<https://groups.google.com/g/google-appengine/c/kAgs8l2_bcE/m/UtXQYMkRAwAJ> I 
was able to find the 500 error. It was on /_ah/warmup and didn't show as an 
error or warning in the logs. I'm not sure why it would have returned an 
error (all that endpoint does is return a 200, so the instance must have 
just failed to boot somehow, but there's nothing useful in the logs).

Now I'm trying to figure out how I could filter out errors on warmup calls, 
since I don't think there's anything I could possibly do to prevent these 
errors. I would have thought that would count as a "loading" response if 
anything did, but I guess not.

On Friday, April 22, 2022 at 12:53:06 PM UTC-4 Alan deLespinasse wrote:

> Hi, I'm trying to debug an alert I have set up. It's a pretty low volume 
> service, App Engine standard environment, Python 3.x.
>
> It's supposed to raise an alert any time there's a 500 response or higher 
> on a query. I've occasionally been getting alerts, but then have been 
> unable to find any 5xx errors in the logs.
>
> The alert configuration looks like this:
> [image: GAEalert1.png]
> ---
> [image: GAEalert2.png]
> I don't remember why I set the function to "rate" instead of "count", but 
> as far as I can tell it shouldn't make a difference. The threshold is zero, 
> so it should trigger if (and, I would hope, only if) there are any 500x 
> errors
>
> I didn't have the "loading = false" filter when I first set it up 
> (sometime last year), but I got a few random alerts when nothing seemed 
> wrong, and the "loading = false" filter seemed to fix it at the time. I was 
> never clear on why (I assume "loading" means that the request was handled 
> by a new instance that was not yet done booting up when the request came 
> in, or something like that).
>
> It's only in the last couple months, I think, that I've started getting 
> these spurious alerts again.
>
> Am I doing something wrong?
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/086cb7a0-8e91-4cbd-96ed-48b6d5c598d3n%40googlegroups.com.

[google-appengine] Alert triggered for 500 responses, but no 500 responses in logs

[Alan deLespinasse]

Hi, I'm trying to debug an alert I have set up. It's a pretty low volume 
service, App Engine standard environment, Python 3.x.

It's supposed to raise an alert any time there's a 500 response or higher 
on a query. I've occasionally been getting alerts, but then have been 
unable to find any 5xx errors in the logs.

The alert configuration looks like this:
[image: GAEalert1.png]
---
[image: GAEalert2.png]
I don't remember why I set the function to "rate" instead of "count", but 
as far as I can tell it shouldn't make a difference. The threshold is zero, 
so it should trigger if (and, I would hope, only if) there are any 500x 
errors

I didn't have the "loading = false" filter when I first set it up (sometime 
last year), but I got a few random alerts when nothing seemed wrong, and 
the "loading = false" filter seemed to fix it at the time. I was never 
clear on why (I assume "loading" means that the request was handled by a 
new instance that was not yet done booting up when the request came in, or 
something like that).

It's only in the last couple months, I think, that I've started getting 
these spurious alerts again.

Am I doing something wrong?

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/d4b1c760-fa3d-4411-9347-36ba28607254n%40googlegroups.com.

[google-appengine] Re: Appengine deployment error

[Alan deLespinasse]

And indeed setting a job_age_limit on the cron job seems to avoid the error 
for me. Still seems like a bug was introduced somewhere.

On Thursday, January 28, 2021 at 6:50:05 PM UTC-5 Alan deLespinasse wrote:

> I think it's this bug: https://issuetracker.google.com/issues/178529949
>
>
> On Thursday, January 28, 2021 at 6:47:25 PM UTC-5 Alan deLespinasse wrote:
>
>> I'm having the same issue. This was working fine until a couple days ago. 
>> The last known successful deploy was January 25 at 17:43 EST. The first 
>> known failure was January 26 at 13:44 EST. Example transcript:
>>
>> > gcloud app --quiet --project gamalon-emailabz-debug deploy cron.yaml
>> Configurations to update:
>>
>> descriptor:  [/home/aldel/emailabz/server/cron.yaml]
>> type:[cron jobs]
>> target project:  [gamalon-emailabz-debug]
>>
>>
>> Updating config [cron]...failed.  
>> 
>>   
>> ERROR: gcloud crashed (TypeError): 'NoneType' object is not subscriptable
>>
>> If you would like to report this issue, please run the following command:
>>   gcloud feedback
>>
>> To check gcloud for common problems, please run the following command:
>>   gcloud info --run-diagnostics
>>
>>
>> The cron.yaml is very simple and has not changed.
>>
>> On Thursday, January 28, 2021 at 1:29:55 PM UTC-5 Elliott (Cloud Platform 
>> Support) wrote:
>>
>>> Hello Jagath,
>>>
>>> The issue that I was referring to was related to deployment of 
>>> applications in general and it was resolved yesterday. It appears that your 
>>> question and scenario may be unrelated and needs to be investigated 
>>> further. What is the actual error you are getting? Please provide a 
>>> redacted version and we will look into this for you. I was under the 
>>> impression that there was a configuration issue in your project and once 
>>> you corrected it, you were able to deploy without a problem. It appears 
>>> that I was mistaken and if this is the case, please tell. Google Groups is 
>>> meant for general discussions but if you want to report a common issue, you 
>>> may use Google Issue Tracker 
>>> <https://issuetracker.google.com/issues/new?component=187191=0>
>>> .
>>>
>>> I will wait for your response and apologies if I misunderstood.
>>>
>>> On Thursday, January 28, 2021 at 12:49:21 PM UTC-5 Jagath Weerasinghe 
>>> wrote:
>>>
>>>> Hello Elliot, 
>>>>
>>>> I guess the relevant engineering teams are aware of this issue, as we 
>>>> can not keep skipping the cron job deployment like this. 
>>>>
>>>> Thanks, 
>>>> Jagath
>>>>
>>>> On Thursday, January 28, 2021 at 6:31:13 PM UTC+1 Elliott (Cloud 
>>>> Platform Support) wrote:
>>>>
>>>>> Hello Jagath,
>>>>>
>>>>> I’m glad that you were able to find the root cause of the deployment 
>>>>> error and now you are able to deploy. You were correct to enable the API 
>>>>> since you apparently needed it in your case.
>>>>>
>>>>> I was able to provide you with the publicly available information 
>>>>> regarding the common issue reported on Tuesday and I hope it is no longer 
>>>>> causing you difficulties.
>>>>>
>>>>> If you have similar issues in the future and wish to have a general 
>>>>> discussion regarding that, you may create another Google Groups thread.
>>>>>
>>>>> Thank you for your patience and I wish you success in your application.
>>>>>
>>>>>
>>>>> On Thursday, January 28, 2021 at 7:35:04 AM UTC-5 Jagath Weerasinghe 
>>>>> wrote:
>>>>>
>>>>>> Hello, 
>>>>>>
>>>>>> Looks like deploying cron (no changes to cron jobs from our last 
>>>>>> deployment) is the issue. I skipped cron and the deployment worked. 
>>>>>> I had to enable cloudtasks.googleapis.com for deploying task queues. 
>>>>>>
>>>>>> Jagath
>>>>>>
>>>>>> On Thursday, January 28, 2021 at 9:37:08 AM UTC+1 Jagath Weerasinghe 
>>>>>> wrote:
>>>>>>
>>>>>>> Hello, 
>>

[google-appengine] Re: Appengine deployment error

[Alan deLespinasse]

I think it's this bug: https://issuetracker.google.com/issues/178529949


On Thursday, January 28, 2021 at 6:47:25 PM UTC-5 Alan deLespinasse wrote:

> I'm having the same issue. This was working fine until a couple days ago. 
> The last known successful deploy was January 25 at 17:43 EST. The first 
> known failure was January 26 at 13:44 EST. Example transcript:
>
> > gcloud app --quiet --project gamalon-emailabz-debug deploy cron.yaml
> Configurations to update:
>
> descriptor:  [/home/aldel/emailabz/server/cron.yaml]
> type:[cron jobs]
> target project:  [gamalon-emailabz-debug]
>
>
> Updating config [cron]...failed.  
> 
>   
> ERROR: gcloud crashed (TypeError): 'NoneType' object is not subscriptable
>
> If you would like to report this issue, please run the following command:
>   gcloud feedback
>
> To check gcloud for common problems, please run the following command:
>   gcloud info --run-diagnostics
>
>
> The cron.yaml is very simple and has not changed.
>
> On Thursday, January 28, 2021 at 1:29:55 PM UTC-5 Elliott (Cloud Platform 
> Support) wrote:
>
>> Hello Jagath,
>>
>> The issue that I was referring to was related to deployment of 
>> applications in general and it was resolved yesterday. It appears that your 
>> question and scenario may be unrelated and needs to be investigated 
>> further. What is the actual error you are getting? Please provide a 
>> redacted version and we will look into this for you. I was under the 
>> impression that there was a configuration issue in your project and once 
>> you corrected it, you were able to deploy without a problem. It appears 
>> that I was mistaken and if this is the case, please tell. Google Groups is 
>> meant for general discussions but if you want to report a common issue, you 
>> may use Google Issue Tracker 
>> <https://issuetracker.google.com/issues/new?component=187191=0>.
>>
>> I will wait for your response and apologies if I misunderstood.
>>
>> On Thursday, January 28, 2021 at 12:49:21 PM UTC-5 Jagath Weerasinghe 
>> wrote:
>>
>>> Hello Elliot, 
>>>
>>> I guess the relevant engineering teams are aware of this issue, as we 
>>> can not keep skipping the cron job deployment like this. 
>>>
>>> Thanks, 
>>> Jagath
>>>
>>> On Thursday, January 28, 2021 at 6:31:13 PM UTC+1 Elliott (Cloud 
>>> Platform Support) wrote:
>>>
>>>> Hello Jagath,
>>>>
>>>> I’m glad that you were able to find the root cause of the deployment 
>>>> error and now you are able to deploy. You were correct to enable the API 
>>>> since you apparently needed it in your case.
>>>>
>>>> I was able to provide you with the publicly available information 
>>>> regarding the common issue reported on Tuesday and I hope it is no longer 
>>>> causing you difficulties.
>>>>
>>>> If you have similar issues in the future and wish to have a general 
>>>> discussion regarding that, you may create another Google Groups thread.
>>>>
>>>> Thank you for your patience and I wish you success in your application.
>>>>
>>>>
>>>> On Thursday, January 28, 2021 at 7:35:04 AM UTC-5 Jagath Weerasinghe 
>>>> wrote:
>>>>
>>>>> Hello, 
>>>>>
>>>>> Looks like deploying cron (no changes to cron jobs from our last 
>>>>> deployment) is the issue. I skipped cron and the deployment worked. 
>>>>> I had to enable cloudtasks.googleapis.com for deploying task queues. 
>>>>>
>>>>> Jagath
>>>>>
>>>>> On Thursday, January 28, 2021 at 9:37:08 AM UTC+1 Jagath Weerasinghe 
>>>>> wrote:
>>>>>
>>>>>> Hello, 
>>>>>>
>>>>>> Thanks for the reply. We have several projects in us-central1 and 
>>>>>> europe-west1 regions. All the projects experience the same issue. 
>>>>>> We are running appengine standard version and we did not experience a 
>>>>>> time-out issue recently. 
>>>>>>
>>>>>> Do you have more information about the issue. For example, whether 
>>>>>> the issue is related only to the cron job deployment, if so, and if we 
>>>>>> skip 
>>>>>

[google-appengine] Re: Appengine deployment error

[Alan deLespinasse]

I'm having the same issue. This was working fine until a couple days ago. 
The last known successful deploy was January 25 at 17:43 EST. The first 
known failure was January 26 at 13:44 EST. Example transcript:

> gcloud app --quiet --project gamalon-emailabz-debug deploy cron.yaml
Configurations to update:

descriptor:  [/home/aldel/emailabz/server/cron.yaml]
type:[cron jobs]
target project:  [gamalon-emailabz-debug]


Updating config [cron]...failed.


ERROR: gcloud crashed (TypeError): 'NoneType' object is not subscriptable

If you would like to report this issue, please run the following command:
  gcloud feedback

To check gcloud for common problems, please run the following command:
  gcloud info --run-diagnostics


The cron.yaml is very simple and has not changed.

On Thursday, January 28, 2021 at 1:29:55 PM UTC-5 Elliott (Cloud Platform 
Support) wrote:

> Hello Jagath,
>
> The issue that I was referring to was related to deployment of 
> applications in general and it was resolved yesterday. It appears that your 
> question and scenario may be unrelated and needs to be investigated 
> further. What is the actual error you are getting? Please provide a 
> redacted version and we will look into this for you. I was under the 
> impression that there was a configuration issue in your project and once 
> you corrected it, you were able to deploy without a problem. It appears 
> that I was mistaken and if this is the case, please tell. Google Groups is 
> meant for general discussions but if you want to report a common issue, you 
> may use Google Issue Tracker 
> .
>
> I will wait for your response and apologies if I misunderstood.
>
> On Thursday, January 28, 2021 at 12:49:21 PM UTC-5 Jagath Weerasinghe 
> wrote:
>
>> Hello Elliot, 
>>
>> I guess the relevant engineering teams are aware of this issue, as we can 
>> not keep skipping the cron job deployment like this. 
>>
>> Thanks, 
>> Jagath
>>
>> On Thursday, January 28, 2021 at 6:31:13 PM UTC+1 Elliott (Cloud Platform 
>> Support) wrote:
>>
>>> Hello Jagath,
>>>
>>> I’m glad that you were able to find the root cause of the deployment 
>>> error and now you are able to deploy. You were correct to enable the API 
>>> since you apparently needed it in your case.
>>>
>>> I was able to provide you with the publicly available information 
>>> regarding the common issue reported on Tuesday and I hope it is no longer 
>>> causing you difficulties.
>>>
>>> If you have similar issues in the future and wish to have a general 
>>> discussion regarding that, you may create another Google Groups thread.
>>>
>>> Thank you for your patience and I wish you success in your application.
>>>
>>>
>>> On Thursday, January 28, 2021 at 7:35:04 AM UTC-5 Jagath Weerasinghe 
>>> wrote:
>>>
 Hello, 

 Looks like deploying cron (no changes to cron jobs from our last 
 deployment) is the issue. I skipped cron and the deployment worked. 
 I had to enable cloudtasks.googleapis.com for deploying task queues. 

 Jagath

 On Thursday, January 28, 2021 at 9:37:08 AM UTC+1 Jagath Weerasinghe 
 wrote:

> Hello, 
>
> Thanks for the reply. We have several projects in us-central1 and 
> europe-west1 regions. All the projects experience the same issue. 
> We are running appengine standard version and we did not experience a 
> time-out issue recently. 
>
> Do you have more information about the issue. For example, whether the 
> issue is related only to the cron job deployment, if so, and if we skip 
> cron job deployment would the deployment still work? Or is there any 
> other 
> workaround?
>
> Thank you, 
> Jagath
>
>
> On Wednesday, January 27, 2021 at 9:22:15 PM UTC+1 Elliott (Cloud 
> Platform Support) wrote:
>
>> Hello
>>
>> Thank you for your question. I know it is frustrating when an app 
>> cannot deploy. There was an issue yesterday for some App Engine Flexible 
>> deployments that failed with a timeout. That issue with Google App 
>> Engine 
>> was resolved for all affected projects as of Tuesday. I hope this did 
>> not 
>> cause inconvenience to you.
>>
>> We appreciate your patience while we worked on resolving the issue.
>>
>> If you have other related questions, please let us know.
>>
>>
>> On Wednesday, January 27, 2021 at 11:18:17 AM UTC-5 Jagath 
>> Weerasinghe wrote:
>>
>>> Hi All, 
>>>
>>> We have been running appengine deployment smoothly till yesterday. 
>>> Yesterday gcloud deploy started to complain about 
>>> *cloudscheduler.googleapis.com 
>>> . *I 

[google-appengine] Re: min_instances, min_idle_instances, and old versions

[Alan deLespinasse]

Note that I've had similar-looking issues 
<https://groups.google.com/g/google-appengine/c/eJUQ7NlNkso/m/3xav-lCEBAAJ> 
in the past, though that was under very different conditions and it 
appeared to get fixed.

On Friday, October 2, 2020 at 11:54:49 AM UTC-4 Alan deLespinasse wrote:

> Not sure what you mean by the default version. There's a default 
> *service*, but generally you have a different configuration file for each 
> service, so it's easy enough to have a different value of min_instances for 
> each one.
>
> My best advice for now is not to use min_instances; use min_idle_instances 
> instead. Unfortunately settings min_idle_instances can result in a larger 
> number of running instances than setting min_instances to the same number. 
> For example, if min_idle_instances is 2, and you have a pretty low level of 
> traffic that could easily be handled by one instance, then you'll have 3 
> instances. Whereas if min_instances were 2, you'd have only 2 instances 
> under the same conditions.
>
> The other option is probably some kind of scripting in your release 
> process to delete instances of old versions, or even to delete all old 
> versions (which I wouldn't want to do, because having old versions still 
> around is very useful if you need to do a quick rollback). I'm not really 
> sure what's possible.
>
> On Friday, October 2, 2020 at 11:35:20 AM UTC-4 Boris Brudnoy wrote:
>
>> Thanks for this conversation, it clarified some matters for me. 
>>
>> Is there, then, a way to tell the App Engine Standard Environment to only 
>> apply the min_instance setting to the *default* app version, especially 
>> if 100% of traffic is now directed at that default version? I'd like to 
>> avoid the scenario of older app versions running unused instances.
>>
>> Thanks,
>>
>> Boris
>>
>> On Monday, September 28, 2020 at 5:43:09 PM UTC-4 adeles...@gmail.com 
>> wrote:
>>
>>> tl;dr: *Never use min_instances!* It will just increase your bill 
>>> unnecessarily.
>>>
>>> On Thursday, September 17, 2020 at 2:26:50 PM UTC-4 Olu wrote:
>>>
>>>>
>>>> To start with, I can confirm that you would be billed for all Instances 
>>>> in use, whether or not they are actively serving requests, traffic or not. 
>>>>
>>>> I will attempt to response to your inquiries as I have highlighted them 
>>>> below:
>>>>
>>>> 1. Is the information shared on this link[1] accurate?
>>>>
>>>> A: It is not exactly clear which part of the information you are 
>>>> looking to verify. However, I assume you are trying to confirm the 
>>>> explanation about min_instance and min_idle_instances. If so, yes, the 
>>>> information is accurate as those words were copied verbatim from the 
>>>> Documentation[2][3]. If not, please reply to this thread. 
>>>>
>>>
>>> Sorry, I guess I was referring to something implied by that link, not 
>>> directly stated, which is that setting min_instances to 1 or more will 
>>> result in instances never getting shut down in old versions, even if they 
>>> are not receiving traffic.
>>>
>>>  
>>>
>>>> 2. if an auto-scaled service has min_instances set to nonzero, does 
>>>> that mean that instances in old versions don't get shut down when you 
>>>> deploy a new version? And those instances get billed?
>>>>
>>>> A: I believe this article[4] explains in detail how Instances are 
>>>> managed, particularly on scaling down. Scaling down Instances depend on 
>>>> the 
>>>> decrease in the request volumes. Typically, App Engine Standard 
>>>> environment 
>>>> scales down to 0[5] and as explained here[6], if the scheduler decides 
>>>> shuts down active instances due to lack of requests being handled, another 
>>>> instance will not start until prompted by an external request, even with 
>>>> the min_instance set.
>>>>
>>>> With all that being said, as explained in this documentation[7], the 
>>>> default behavior of App Engine Standard is that whenever a new application 
>>>> version is deployed, except the --no-promote flag is used in the 
>>>> deployment, the newly deployed version is automatically configured to 
>>>> receive 100% of traffic. So, with no traffic to the older version, the 
>>>> scheduler would shut down the instances due to lack of requests, even if 
>>>> the min_instance is set to nonzero.
>>>>
>>

[google-appengine] Re: min_instances, min_idle_instances, and old versions

[Alan deLespinasse]

Not sure what you mean by the default version. There's a default *service*, 
but generally you have a different configuration file for each service, so 
it's easy enough to have a different value of min_instances for each one.

My best advice for now is not to use min_instances; use min_idle_instances 
instead. Unfortunately settings min_idle_instances can result in a larger 
number of running instances than setting min_instances to the same number. 
For example, if min_idle_instances is 2, and you have a pretty low level of 
traffic that could easily be handled by one instance, then you'll have 3 
instances. Whereas if min_instances were 2, you'd have only 2 instances 
under the same conditions.

The other option is probably some kind of scripting in your release process 
to delete instances of old versions, or even to delete all old versions 
(which I wouldn't want to do, because having old versions still around is 
very useful if you need to do a quick rollback). I'm not really sure what's 
possible.

On Friday, October 2, 2020 at 11:35:20 AM UTC-4 Boris Brudnoy wrote:

> Thanks for this conversation, it clarified some matters for me. 
>
> Is there, then, a way to tell the App Engine Standard Environment to only 
> apply the min_instance setting to the *default* app version, especially 
> if 100% of traffic is now directed at that default version? I'd like to 
> avoid the scenario of older app versions running unused instances.
>
> Thanks,
>
> Boris
>
> On Monday, September 28, 2020 at 5:43:09 PM UTC-4 adeles...@gmail.com 
> wrote:
>
>> tl;dr: *Never use min_instances!* It will just increase your bill 
>> unnecessarily.
>>
>> On Thursday, September 17, 2020 at 2:26:50 PM UTC-4 Olu wrote:
>>
>>>
>>> To start with, I can confirm that you would be billed for all Instances 
>>> in use, whether or not they are actively serving requests, traffic or not. 
>>>
>>> I will attempt to response to your inquiries as I have highlighted them 
>>> below:
>>>
>>> 1. Is the information shared on this link[1] accurate?
>>>
>>> A: It is not exactly clear which part of the information you are looking 
>>> to verify. However, I assume you are trying to confirm the explanation 
>>> about min_instance and min_idle_instances. If so, yes, the information is 
>>> accurate as those words were copied verbatim from the Documentation[2][3]. 
>>> If not, please reply to this thread. 
>>>
>>
>> Sorry, I guess I was referring to something implied by that link, not 
>> directly stated, which is that setting min_instances to 1 or more will 
>> result in instances never getting shut down in old versions, even if they 
>> are not receiving traffic.
>>
>>  
>>
>>> 2. if an auto-scaled service has min_instances set to nonzero, does that 
>>> mean that instances in old versions don't get shut down when you deploy a 
>>> new version? And those instances get billed?
>>>
>>> A: I believe this article[4] explains in detail how Instances are 
>>> managed, particularly on scaling down. Scaling down Instances depend on the 
>>> decrease in the request volumes. Typically, App Engine Standard environment 
>>> scales down to 0[5] and as explained here[6], if the scheduler decides 
>>> shuts down active instances due to lack of requests being handled, another 
>>> instance will not start until prompted by an external request, even with 
>>> the min_instance set.
>>>
>>> With all that being said, as explained in this documentation[7], the 
>>> default behavior of App Engine Standard is that whenever a new application 
>>> version is deployed, except the --no-promote flag is used in the 
>>> deployment, the newly deployed version is automatically configured to 
>>> receive 100% of traffic. So, with no traffic to the older version, the 
>>> scheduler would shut down the instances due to lack of requests, even if 
>>> the min_instance is set to nonzero.
>>>
>>
>> Obviously setting min_instances overrides the default behavior of scaling 
>> down to zero. And I'm now convinced that, with min_instances set to 
>> nonzero, it doesn't scale down to zero even in obsolete versions that are 
>> set to receive no traffic. This isn't documented behavior, but I've seen it 
>> implied elsewhere (like the Server Fault page above), and it was more or 
>> less confirmed by the agent who handled my billing complaint (they checked 
>> with support engineers, I believe).
>>
>> (As the documentation mentions, "For this feature to function properly, 
>> you must make sure that warmup requests are enabled and that your 
>> application handles warmup requests." So some users may have min_instances 
>> set to more than zero, but not see the above problem, because it is not 
>> actually configured correctly to maintain a minimum number of instances. I 
>> made this mistake for a while.)
>>
>>  
>>
>>> If you are experiencing a different behavior, I suggest you reach out 
>>> directly to the GCP Support Engineers[8] for better evaluation of the 
>>> issue. 
>>>
>>
>> So apparently 

[google-appengine] Re: min_instances, min_idle_instances, and old versions

[Alan deLespinasse]

tl;dr: *Never use min_instances!* It will just increase your bill 
unnecessarily.

On Thursday, September 17, 2020 at 2:26:50 PM UTC-4 Olu wrote:

>
> To start with, I can confirm that you would be billed for all Instances in 
> use, whether or not they are actively serving requests, traffic or not. 
>
> I will attempt to response to your inquiries as I have highlighted them 
> below:
>
> 1. Is the information shared on this link[1] accurate?
>
> A: It is not exactly clear which part of the information you are looking 
> to verify. However, I assume you are trying to confirm the explanation 
> about min_instance and min_idle_instances. If so, yes, the information is 
> accurate as those words were copied verbatim from the Documentation[2][3]. 
> If not, please reply to this thread. 
>

Sorry, I guess I was referring to something implied by that link, not 
directly stated, which is that setting min_instances to 1 or more will 
result in instances never getting shut down in old versions, even if they 
are not receiving traffic.

 

> 2. if an auto-scaled service has min_instances set to nonzero, does that 
> mean that instances in old versions don't get shut down when you deploy a 
> new version? And those instances get billed?
>
> A: I believe this article[4] explains in detail how Instances are managed, 
> particularly on scaling down. Scaling down Instances depend on the decrease 
> in the request volumes. Typically, App Engine Standard environment scales 
> down to 0[5] and as explained here[6], if the scheduler decides shuts down 
> active instances due to lack of requests being handled, another instance 
> will not start until prompted by an external request, even with the 
> min_instance set.
>
> With all that being said, as explained in this documentation[7], the 
> default behavior of App Engine Standard is that whenever a new application 
> version is deployed, except the --no-promote flag is used in the 
> deployment, the newly deployed version is automatically configured to 
> receive 100% of traffic. So, with no traffic to the older version, the 
> scheduler would shut down the instances due to lack of requests, even if 
> the min_instance is set to nonzero.
>

Obviously setting min_instances overrides the default behavior of scaling 
down to zero. And I'm now convinced that, with min_instances set to 
nonzero, it doesn't scale down to zero even in obsolete versions that are 
set to receive no traffic. This isn't documented behavior, but I've seen it 
implied elsewhere (like the Server Fault page above), and it was more or 
less confirmed by the agent who handled my billing complaint (they checked 
with support engineers, I believe).

(As the documentation mentions, "For this feature to function properly, you 
must make sure that warmup requests are enabled and that your application 
handles warmup requests." So some users may have min_instances set to more 
than zero, but not see the above problem, because it is not actually 
configured correctly to maintain a minimum number of instances. I made this 
mistake for a while.)

 

> If you are experiencing a different behavior, I suggest you reach out 
> directly to the GCP Support Engineers[8] for better evaluation of the 
> issue. 
>

So apparently I have to pay for a support plan just to get information that 
should be in the documentation...

 

> 3. What is min_idle_instances actually supposed to mean?
>
> A: As explained in the documentation[3], this is the number of instances 
> that keeps running and ready to serve traffic. The idle Instances helps to 
> avoid the effect of pending latency on your App Engine application.
>

I noticed that this documentation has recently been updated (maybe partly 
in response to my complaints?). It now says "The number of *additional* 
instances..." (emphasis mine), and goes on to explain that by "additional 
instances", it means that App Engine calculates the "necessary" number of 
instances to server current load, and adds on min_idle_instances more 
instances. So it does *not* mean that there will always be this many "idle" 
instances (for some definition of "idle"), as the name would imply. The new 
documentation is a big improvement. (new version 

 
/ old version 

)

(Still waiting for the min_instances documentation to be updated to warn 
about the danger of zombie instances)

 

> As I may have alluded above, Instances are created whenever requests are 
> received. When instances are created, there are certain steps that apply 
> for the Instance to start up and be ready to attend to requests. These are 
> explained in these documentation[9][10]. Basically, having Idle instances 
> help to avoid such steps that would cause pending latency.
>
> 4.  Do I need to set max_idle_instances? 
>
> A: 

[google-appengine] min_instances, min_idle_instances, and old versions

[Alan deLespinasse]

First question: Is this accurate ? That 
is, if an auto-scaled service has min_instances set to nonzero, does that 
mean that instances in old versions don't get shut down when you deploy a 
new version? And those instances get billed?

I've been running a service with the following configuration (standard 
environment, Python 3.7):

runtime: python37
instance_class: F4
automatic_scaling:
  min_instances: 1
  max_instances: 10
inbound_services:
- warmup

New versions are deployed frequently because it's our integration 
environment. Apparently we've ended up with more and more instances 
running, because when we deploy a new version, the old version continues to 
exist and have 1 running instance. And apparently we're getting billed for 
these. I don't think I should have expected this, based on a close reading 
of the documentation 
.
 
(I have opened a billing support request, since I think it's Google's error 
in documentation, if not actually a bug.)

So now I'm trying to fix the configuration to avoid this. Based on the 
Server Fault article I linked above, I tried setting min_instances to 0 and 
min_idle_instances to 1. This seems to result in always at least 2 
instances running. I think maybe because one instance is getting requests 
(we have a once-a-minute cron job, among other things), so it's not "idle", 
so there has to be one more instance to have a minimum of one idle instance.

So I tried setting both min_instances and min_idle_instances to 0, but I 
*still* seem to always have at least 2 instances.

It's really hard to tell though, because the GCP console sometimes takes a 
bit to update, and maybe sometimes there are actually more instances than 
the configuration requires (I think maybe they're not always billed?).

So, second question: What is min_idle_instances actually supposed to mean? 
Is it the minimum number of instances, or is it the minimum number of 
"idle" instances, for some definition of "idle"? If an instance is serving 
1 simple query per minute, does that mean it's not idle, so there will be a 
second, idle instance? But then there are 2 instances, and if there are a 
few simple queries happening per minute, it seems like some might be 
randomly routed to each instance, so neither instance would be idle, and a 
third instance would get started.

Another complication: in our production environment, I increased 
min_instances to 2 because of this issue 
. 
It's pretty important that we always (>99.9% anyway) have at least 1 
running instance, since apparently there's no way to get instance startup 
time to less than 20-30 seconds, and to guarantee that, apparently we need 
2 instances running most of the time, because they can get preempted at any 
time without warming up a replacement first. So now I'm not sure whether to 
set min_idle_instances to 1 or 2 or what in production.

Do I need to set max_idle_instances? The documentation says its default 
value is "automatic", but doesn't say what that actually means.

I'm having a hard time figuring out these issues based on the documentation.

All I want is

   - Instances get shut down when I deploy a new version (I would have 
   thought this was always the case no matter what!)
   - Each of my environments normally just has 1 instance running, assuming 
   light traffic (1-10 queries per minute). Having 2 instances always running 
   in production is ok, if that's the only way to achieve the next point:
   - Never (or almost never) have zero instances running (aka almost never 
   have a query take more than 2 seconds because of warmup time)
   - Autoscale up to a reasonable maximum if traffic gets heavier

I didn't think this would be so difficult. I've been using App Engine for a 
long time, and thought I knew what I was doing, but I guess I've never used 
these options in the current environment (Python 3, standard).

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/84d8f642-3eae-4144-962e-71a1990dbbedn%40googlegroups.com.

[google-appengine] Re: Instance gets terminated without warming up another

[Alan deLespinasse]

Possibly relevant, but basically comes to the conclusion of "increase the 
minimum instances": 
https://stackoverflow.com/questions/57054132/improving-cold-start-up-times-on-google-app-engine-running-django-on-python-3-7

Also, is there any way to tell why the instance was terminated? There's 
nothing in the logs that makes it clear to me. I've been assuming that it 
was being preempted by some other process. For some reason it seems to 
happen a lot more frequently in our production environment than our staging 
environment, 

On Wednesday, August 12, 2020 at 11:54:16 PM UTC-4, Alan deLespinasse wrote:
>
> Hi, thanks for your response!
>
> I would love to decrease the startup time. I assumed there wasn't any way 
> I could decrease it much, since I'm basically just running a simple Flask 
> server with mostly default app configuration. It does have some third-party 
> package dependencies, but I don't know of any reason why they'd be that 
> slow to load. When I run it locally for development, it starts up in less 
> than a second. (I don't use gunicorn locally; maybe I'll try that and see 
> if it's slow.)
>
> It's been a while since I ran a simple Standard Environment Python app 
> with automatic scaling, and when I did it was probably old-school Python 
> 2.x. I think they used to start up pretty quickly, but I didn't know if it 
> would be the same in newer environments.
>
> I'm on vacation this week, but maybe next week I'll experiment with adding 
> some extra logging before any dependencies are loaded, and when Flask 
> actually starts serving. It would be interesting to see the timing on that.
>
> I could play with gunicorn configuration, but I assumed the defaults were 
> reasonable.
>
> Meanwhile, my coworkers and I actually decided it was a good idea to 
> increase the minimum instances anyway, so this isn't really bothering us 
> now, but it would be nice to solve the mystery. 
>
> Thanks!
>
> Alan
>
> On Wednesday, August 12, 2020 at 9:00:55 PM UTC-4, Elliott (Cloud Platform 
> Support) wrote:
>>
>> Hello Alan,
>>
>> As you have pointed out from the documentation 
>> <https://cloud.google.com/appengine/docs/standard/python3/how-instances-are-managed>:
>>  
>> (I know you’ve read it but please bear with me.)
>>
>> App Engine attempts to keep manual and basic scaling instances running 
>> indefinitely. However, at this time there is no guaranteed uptime for 
>> manual and basic scaling instances. Hardware and software failures that 
>> cause early termination or frequent restarts can occur without prior 
>> warning and can take considerable time to resolve; thus, you should 
>> construct your application in a way that tolerates these failures.
>>
>> *You would like to know if App Engine can resist shutting down the 
>> instance before spinning up a new one. This information is not available in 
>> public documentation and from what is happening in your scenario, I believe 
>> I would have to reach out to an App Engine Specialist to confirm. I want to 
>> make sure.
>>
>> You have provided a workaround to this behavior to set up more instances 
>> but you would like to know what are your other options. (Again, please bear 
>> with me.)
>>
>> From the documentation I included 
>> <https://cloud.google.com/appengine/docs/standard/python3/how-instances-are-managed>
>>  
>> you may:
>>
>> Reduce the amount of time it takes for your instances restart or for new 
>> ones to start.
>>
>> For long-running computations, periodically create checkpoints so that 
>> you can resume from that state.
>>
>> Your app should be "stateless" so that nothing is stored on the instance.
>>
>> Use queues for performing asynchronous task execution.
>>
>> If you configure your instances to manual scaling:
>> Use load balancing across multiple instances.
>>
>> Configure more instances than required to handle normal traffic.
>> Write fall-back logic that uses cached results when a manual scaling 
>> instance is unavailable.
>>
>> So to summarize right now with the information I have, you have two 
>> options. The first and more direct is to create another instance that would 
>> affect your budget and the other to look at the advice for your application.
>>
>> You also have another issue where it may take 30 seconds to warm up an 
>> instance. This information is private and would require looking at your 
>> project. I can help you as much as I can using this forum but sharing 
>> specific information about your project is not good for security.
>>
>> I will wait for your response.
>>
>>
>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/613628f8-1a18-43fa-9694-1fe494a8a5d3o%40googlegroups.com.

[google-appengine] Re: Instance gets terminated without warming up another

[Alan deLespinasse]

Hi, thanks for your response!

I would love to decrease the startup time. I assumed there wasn't any way I 
could decrease it much, since I'm basically just running a simple Flask 
server with mostly default app configuration. It does have some third-party 
package dependencies, but I don't know of any reason why they'd be that 
slow to load. When I run it locally for development, it starts up in less 
than a second. (I don't use gunicorn locally; maybe I'll try that and see 
if it's slow.)

It's been a while since I ran a simple Standard Environment Python app with 
automatic scaling, and when I did it was probably old-school Python 2.x. I 
think they used to start up pretty quickly, but I didn't know if it would 
be the same in newer environments.

I'm on vacation this week, but maybe next week I'll experiment with adding 
some extra logging before any dependencies are loaded, and when Flask 
actually starts serving. It would be interesting to see the timing on that.

I could play with gunicorn configuration, but I assumed the defaults were 
reasonable.

Meanwhile, my coworkers and I actually decided it was a good idea to 
increase the minimum instances anyway, so this isn't really bothering us 
now, but it would be nice to solve the mystery. 

Thanks!

Alan

On Wednesday, August 12, 2020 at 9:00:55 PM UTC-4, Elliott (Cloud Platform 
Support) wrote:
>
> Hello Alan,
>
> As you have pointed out from the documentation 
> :
>  
> (I know you’ve read it but please bear with me.)
>
> App Engine attempts to keep manual and basic scaling instances running 
> indefinitely. However, at this time there is no guaranteed uptime for 
> manual and basic scaling instances. Hardware and software failures that 
> cause early termination or frequent restarts can occur without prior 
> warning and can take considerable time to resolve; thus, you should 
> construct your application in a way that tolerates these failures.
>
> *You would like to know if App Engine can resist shutting down the 
> instance before spinning up a new one. This information is not available in 
> public documentation and from what is happening in your scenario, I believe 
> I would have to reach out to an App Engine Specialist to confirm. I want to 
> make sure.
>
> You have provided a workaround to this behavior to set up more instances 
> but you would like to know what are your other options. (Again, please bear 
> with me.)
>
> From the documentation I included 
> 
>  
> you may:
>
> Reduce the amount of time it takes for your instances restart or for new 
> ones to start.
>
> For long-running computations, periodically create checkpoints so that you 
> can resume from that state.
>
> Your app should be "stateless" so that nothing is stored on the instance.
>
> Use queues for performing asynchronous task execution.
>
> If you configure your instances to manual scaling:
> Use load balancing across multiple instances.
>
> Configure more instances than required to handle normal traffic.
> Write fall-back logic that uses cached results when a manual scaling 
> instance is unavailable.
>
> So to summarize right now with the information I have, you have two 
> options. The first and more direct is to create another instance that would 
> affect your budget and the other to look at the advice for your application.
>
> You also have another issue where it may take 30 seconds to warm up an 
> instance. This information is private and would require looking at your 
> project. I can help you as much as I can using this forum but sharing 
> specific information about your project is not good for security.
>
> I will wait for your response.
>
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/f8bd681b-d7ae-4596-aba0-1fe39094cb02o%40googlegroups.com.

[google-appengine] Re: Instance gets terminated without warming up another

[Alan deLespinasse]

I'm also not clear on why it would take on the order of 30 seconds to warm 
up an instance, but I assume it's an unavoidable property of GAE. There's 
nothing in my code that would make it take so long, unless it has to 
install the dependencies every time or something.

On Monday, August 3, 2020 at 1:26:37 PM UTC-4, Alan deLespinasse wrote:
>
> Python 3.7, Standard Environment. Here's my entire app.yaml:
>
> runtime: python37
> instance_class: F4
> automatic_scaling:
>  min_instances: 1
>  max_instances: 10
>
> inbound_services:
> - warmup
>
>
>
> The problem: Several times a day, the instance (of which there's only one) 
> seems to get terminated randomly. This is fine; I understand that App 
> Engine doesn't guarantee a single instance will last forever. The problem 
> is that the new instance doesn't start warming up until after the old one 
> stops, which means that any real requests made during warmup end up having 
> very high latency, sometimes over 30 seconds. Shouldn't GAE wait to 
> terminate the old instance until the replacement is finished warming up?
>
> I'm pretty sure the process isn't crashing or running out of memory or 
> anything. Most of the time, this service just handles one request every 
> minute (it's a cron job), and the terminate signals seem to happen at 
> random times in between those requests.
>
> Maybe there's just something I don't understand about GAE configuration.
>
> I could increase min_instances to 2, of course, which I assume would make 
> this much less likely, but I'd rather not. This isn't a high-traffic or 
> (currently) mission critical server. On the other hand, having occasional 
> requests take 30+ seconds can be pretty annoying.
>
> Is there something else I can do?
>
> Yes, I've read How Instances Are Managed 
> <https://cloud.google.com/appengine/docs/standard/python3/how-instances-are-managed>
> .
>
> Typical logs are below (stdout, stderr, and request_log, all versions). 
> You can see the cron job at the top at 22:39:00, 22:40:00, 22:41:00.
>
> At 22:41:47 is the terminate signal, and at 22:41:48 the warmup request 
> for the new instance. There's an exception right after that that I assume 
> is triggered by the process terminating (related to a Firestore connection, 
> I think).
>
> Then at 22:42:00, the next cron request happens, and I think it actually 
> triggers a second new instance because there's no instance currently ready 
> to go. If it just waited for the first one to finish warming up, this 
> request would have taken about 19 seconds. But since it instead starts a 
> second one, it takes 34 seconds, and increases my bill as well.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/2d474331-bf78-4ac2-8f21-38a8960d888bo%40googlegroups.com.

[google-appengine] Re: [Google Cloud Insiders] URGENT: is GAE Standard memcache down????

[Alan deLespinasse]

I also noticed that my failed instances were still running; see my other 
thread 
<https://groups.google.com/forum/#!topic/google-appengine/dOUfeztkcws> for 
details. This could have cost my company money if I hadn't noticed it.

On Tuesday, November 7, 2017 at 11:18:10 AM UTC-5, Yannick (Cloud Platform 
Support) wrote:
>
> This incident was resolved yesterday. See the incident page 
> <https://status.cloud.google.com/incident/appengine/17007> for a timeline.
>
> On Monday, November 6, 2017 at 6:13:56 PM UTC-5, Alan deLespinasse wrote:
>>
>> I maybe should mention that the old version of my service was working 
>> fine, as far as I can tell. I just can't deploy a new version.
>>
>> On Monday, November 6, 2017 at 6:12:00 PM UTC-5, Alan deLespinasse wrote:
>>>
>>> I'm also unable to deploy a "vm: true" service. My app doesn't even use 
>>> memcache, but maybe "vm: true" depends on memcache somehow?
>>>
>>> The error message on my "gcloud app deploy" command said "Timed out when 
>>> starting VMs." I can' t find anything in the logs that means anything to me.
>>>
>>> On Monday, November 6, 2017 at 5:32:00 PM UTC-5, Justin Yip wrote:
>>>>
>>>> We cannot bring up any "vm: true" for the past couple of hours too.
>>>>
>>>> (Maybe the memcache service is also on "vm: true", which all gets 
>>>> turned down :)
>>>>
>>>>
>>>>
>>>> On Monday, November 6, 2017 at 1:27:34 PM UTC-8, Moe Adham wrote:
>>>>>
>>>>> Something pretty catastrophic is going on here, all our services on 
>>>>> app-engine got a stop request and we can't bring them back up.
>>>>>
>>>>> On Monday, November 6, 2017 at 4:11:12 PM UTC-5, PK wrote:
>>>>>>
>>>>>> Yes it is here: https://issuetracker.google.com/issues/68946028
>>>>>>
>>>>>> I also see now a message in the Google Cloud Status Console about 
>>>>>> this.
>>>>>>
>>>>>> On Nov 6, 2017, at 1:07 PM, Wade Piotrowski <wa...@google.com> wrote:
>>>>>>
>>>>>> Yes, Posting shortly. Please file a support case for the most 
>>>>>> up-to-date information if you haven't already.
>>>>>>
>>>>>>
>>>>>> On Mon, Nov 6, 2017 at 12:56 PM PK <p...@gae123.com> wrote:
>>>>>>
>>>>>>> This started returning False 15 minutes ago. My app is down because 
>>>>>>> of it.
>>>>>>>
>>>>>>> from google.appengine.api.capabilities import CapabilitySet
>>>>>>> print CapabilitySet('memcache', methods=['set']).is_enabled()
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google 
>>>>>>> Groups "Google Cloud Insiders" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it, 
>>>>>>> send an email to google-cloud-insiders+unsubscr...@googlegroups.com.
>>>>>>> To post to this group, send email to google-clo...@googlegroups.com.
>>>>>>> Visit this group at 
>>>>>>> https://groups.google.com/group/google-cloud-insiders.
>>>>>>> To view this discussion on the web visit 
>>>>>>> https://groups.google.com/d/msgid/google-cloud-insiders/C170C4A6-5355-428B-B010-4507B5148F3C%40gae123.com
>>>>>>> .
>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>
>>>>>>
>>>>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/35bc9c83-2843-4389-8d02-a411d9fefa6e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Failed to deploy with "vm: true"? Check your versions

[Alan deLespinasse]

Yesterday I had a couple failures when trying to deploy a new version of my 
service, which is configured with "vm: true". It seemed to correlate with 
the memcache outage (even though I don't use memcache); after that was 
fixed I had no trouble deploying.

I happened to look at the list of running versions of my service after 
that, and there were still instances running for each failed attempt to 
deploy. All traffic was still going to the previous, working instance, so 
nothing was broken, but I had to shut down those instances myself.

So, if you had any failed deployments yesterday, you might want to check 
that out. It could cost you money. GCP console -> App Engine -> Versions 
(and select the relevant service if you have more than one).

Hopefully some Google SREs are also on this and will check for extra 
instances that need to be shut down too...

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/ba6eea87-ca64-45f5-be19-1e9f609f23b7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: [Google Cloud Insiders] URGENT: is GAE Standard memcache down????

[Alan deLespinasse]

I maybe should mention that the old version of my service was working fine, 
as far as I can tell. I just can't deploy a new version.

On Monday, November 6, 2017 at 6:12:00 PM UTC-5, Alan deLespinasse wrote:
>
> I'm also unable to deploy a "vm: true" service. My app doesn't even use 
> memcache, but maybe "vm: true" depends on memcache somehow?
>
> The error message on my "gcloud app deploy" command said "Timed out when 
> starting VMs." I can' t find anything in the logs that means anything to me.
>
> On Monday, November 6, 2017 at 5:32:00 PM UTC-5, Justin Yip wrote:
>>
>> We cannot bring up any "vm: true" for the past couple of hours too.
>>
>> (Maybe the memcache service is also on "vm: true", which all gets turned 
>> down :)
>>
>>
>>
>> On Monday, November 6, 2017 at 1:27:34 PM UTC-8, Moe Adham wrote:
>>>
>>> Something pretty catastrophic is going on here, all our services on 
>>> app-engine got a stop request and we can't bring them back up.
>>>
>>> On Monday, November 6, 2017 at 4:11:12 PM UTC-5, PK wrote:
>>>>
>>>> Yes it is here: https://issuetracker.google.com/issues/68946028
>>>>
>>>> I also see now a message in the Google Cloud Status Console about this.
>>>>
>>>> On Nov 6, 2017, at 1:07 PM, Wade Piotrowski <wa...@google.com> wrote:
>>>>
>>>> Yes, Posting shortly. Please file a support case for the most 
>>>> up-to-date information if you haven't already.
>>>>
>>>>
>>>> On Mon, Nov 6, 2017 at 12:56 PM PK <p...@gae123.com> wrote:
>>>>
>>>>> This started returning False 15 minutes ago. My app is down because of 
>>>>> it.
>>>>>
>>>>> from google.appengine.api.capabilities import CapabilitySet
>>>>> print CapabilitySet('memcache', methods=['set']).is_enabled()
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google 
>>>>> Groups "Google Cloud Insiders" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send 
>>>>> an email to google-cloud-insiders+unsubscr...@googlegroups.com.
>>>>> To post to this group, send email to google-clo...@googlegroups.com.
>>>>> Visit this group at 
>>>>> https://groups.google.com/group/google-cloud-insiders.
>>>>> To view this discussion on the web visit 
>>>>> https://groups.google.com/d/msgid/google-cloud-insiders/C170C4A6-5355-428B-B010-4507B5148F3C%40gae123.com
>>>>> .
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>
>>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/ce9c4734-e360-482f-a7e5-6934550b805a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: [Google Cloud Insiders] URGENT: is GAE Standard memcache down????

[Alan deLespinasse]

I'm also unable to deploy a "vm: true" service. My app doesn't even use 
memcache, but maybe "vm: true" depends on memcache somehow?

The error message on my "gcloud app deploy" command said "Timed out when 
starting VMs." I can' t find anything in the logs that means anything to me.

On Monday, November 6, 2017 at 5:32:00 PM UTC-5, Justin Yip wrote:
>
> We cannot bring up any "vm: true" for the past couple of hours too.
>
> (Maybe the memcache service is also on "vm: true", which all gets turned 
> down :)
>
>
>
> On Monday, November 6, 2017 at 1:27:34 PM UTC-8, Moe Adham wrote:
>>
>> Something pretty catastrophic is going on here, all our services on 
>> app-engine got a stop request and we can't bring them back up.
>>
>> On Monday, November 6, 2017 at 4:11:12 PM UTC-5, PK wrote:
>>>
>>> Yes it is here: https://issuetracker.google.com/issues/68946028
>>>
>>> I also see now a message in the Google Cloud Status Console about this.
>>>
>>> On Nov 6, 2017, at 1:07 PM, Wade Piotrowski  wrote:
>>>
>>> Yes, Posting shortly. Please file a support case for the most up-to-date 
>>> information if you haven't already.
>>>
>>>
>>> On Mon, Nov 6, 2017 at 12:56 PM PK  wrote:
>>>
 This started returning False 15 minutes ago. My app is down because of 
 it.

 from google.appengine.api.capabilities import CapabilitySet
 print CapabilitySet('memcache', methods=['set']).is_enabled()



 --
 You received this message because you are subscribed to the Google 
 Groups "Google Cloud Insiders" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to google-cloud-insiders+unsubscr...@googlegroups.com.
 To post to this group, send email to google-clo...@googlegroups.com.
 Visit this group at 
 https://groups.google.com/group/google-cloud-insiders.
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/google-cloud-insiders/C170C4A6-5355-428B-B010-4507B5148F3C%40gae123.com
 .
 For more options, visit https://groups.google.com/d/optout.

>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/36c18448-8a30-445e-bd0d-4b4f4a3363ca%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Old versions still running (flexible environment)

[Alan deLespinasse]

Flexible, Node.js 6.9.3 or something.

On Sunday, July 16, 2017 at 4:04:17 AM UTC-4, Matthew Kime wrote:
>
> Which version of app engine are you using? We're using the flex 
> environment with custom runtime.
>
> thanks,
> matt
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/919a86ba-73f7-464e-b4c1-3ee48b16fdfe%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Old versions still running (flexible environment)

[Alan deLespinasse]

I did have one case a couple weeks ago where an old version was 
unexpectedly still running. So I guess it's not 100% fixed on Google's 
side, although that was the only time I've seen it since January.

On Friday, July 14, 2017 at 6:57:00 PM UTC-4, Matthew Kime wrote:
>
> Have your problems been fixed? I'm still seeing them.
>
> On Monday, March 20, 2017 at 11:35:08 AM UTC-5, Alan deLespinasse wrote:
>>
>> Thanks! I'll wait and see if it happens again.
>>
>> On Monday, March 20, 2017 at 12:28:04 PM UTC-4, Zachary Fewtrell wrote:
>>>
>>> Hi Alan,
>>>  Generally we would expect a failed deployment to rollback (i.e. delete 
>>> the new version & leave the old one running).  However your example is from 
>>> January and some rollback related issues have been addressed since then. 
>>>  If you are still getting the issue I recommend you file an issue in our 
>>> tracker <http://code.google.com/p/googleappengine/issues/list>.
>>>
>>> Regards,
>>>  Zach Fewtrell
>>>
>>> On Thursday, March 16, 2017 at 12:10:07 PM UTC-7, Alan deLespinasse 
>>> wrote:
>>>>
>>>> I'm not sure if this is a bug or expected behavior. It's not what I 
>>>> expected, but I'm still somewhat confused by the Cloud Console.
>>>>
>>>> I have several services for which I noticed there were still some old 
>>>> versions running, although the old versions were serving 0% of traffic. 
>>>> Example:
>>>>
>>>>
>>>> <https://lh3.googleusercontent.com/-YZWAxJMW7x4/WMrgCXinq6I/Mac/iRAXXz_vhjYU9072hPy7ZXGENs4bvoCfQCLcB/s1600/oldservices.png>
>>>>
>>>> The two selected versions were made obsolete by the top version. 
>>>> Normally when I deploy a new version, the previously running version is 
>>>> shut down. In these cases, I think a new version never succeeded in 
>>>> starting up, because I had a bug that would simply crash every time. So 
>>>> here's what apparently happens:
>>>>
>>>>1. Version A is running
>>>>2. I try to deploy version B; it repeatedly fails to start up. My 
>>>>"gcloud app deploy" command eventually returns an error, telling me 
>>>> that 
>>>>deployment failed. Version A is still serving all traffic. Version B is 
>>>>apparently still running, though I didn't know that until just now.
>>>>3. I fix the bug and deploy version C. It starts up, version A is 
>>>>correctly stopped. Apparently version B is still running, though.
>>>>
>>>> So, is this a bug or expected? Is it costing us money? I'm finding it a 
>>>> bit hard to tell what our recent charges are for, although they are a bit 
>>>> higher than expected (still looking into that). I'd submit a billing 
>>>> ticket 
>>>> if I was sure.
>>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/5b4324c1-2ef6-4970-a62e-f14cd6d16588%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Old versions still running (flexible environment)

[Alan deLespinasse]

Thanks! I'll wait and see if it happens again.

On Monday, March 20, 2017 at 12:28:04 PM UTC-4, Zachary Fewtrell wrote:
>
> Hi Alan,
>  Generally we would expect a failed deployment to rollback (i.e. delete 
> the new version & leave the old one running).  However your example is from 
> January and some rollback related issues have been addressed since then. 
>  If you are still getting the issue I recommend you file an issue in our 
> tracker <http://code.google.com/p/googleappengine/issues/list>.
>
> Regards,
>  Zach Fewtrell
>
> On Thursday, March 16, 2017 at 12:10:07 PM UTC-7, Alan deLespinasse wrote:
>>
>> I'm not sure if this is a bug or expected behavior. It's not what I 
>> expected, but I'm still somewhat confused by the Cloud Console.
>>
>> I have several services for which I noticed there were still some old 
>> versions running, although the old versions were serving 0% of traffic. 
>> Example:
>>
>>
>> <https://lh3.googleusercontent.com/-YZWAxJMW7x4/WMrgCXinq6I/Mac/iRAXXz_vhjYU9072hPy7ZXGENs4bvoCfQCLcB/s1600/oldservices.png>
>>
>> The two selected versions were made obsolete by the top version. Normally 
>> when I deploy a new version, the previously running version is shut down. 
>> In these cases, I think a new version never succeeded in starting up, 
>> because I had a bug that would simply crash every time. So here's what 
>> apparently happens:
>>
>>1. Version A is running
>>2. I try to deploy version B; it repeatedly fails to start up. My 
>>"gcloud app deploy" command eventually returns an error, telling me that 
>>deployment failed. Version A is still serving all traffic. Version B is 
>>apparently still running, though I didn't know that until just now.
>>3. I fix the bug and deploy version C. It starts up, version A is 
>>correctly stopped. Apparently version B is still running, though.
>>
>> So, is this a bug or expected? Is it costing us money? I'm finding it a 
>> bit hard to tell what our recent charges are for, although they are a bit 
>> higher than expected (still looking into that). I'd submit a billing ticket 
>> if I was sure.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/85c9513b-be7b-4b4e-9aaf-dfc8f65c737e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Old versions still running (flexible environment)

[Alan deLespinasse]

We have some services that are manual and some that are automatic. I'm 
pretty sure both have had this problem.

I wouldn't have interpreted that documentation as saying that manual 
scaling would keep old *versions* running. Our manual-scaling services are 
configured to have one *instance*. We've ended up having multiple *versions* 
running, 
each with one instance, although all traffic was routed to a single version.

And like I said, normally when we deploy a new version, the old version is 
shut down automatically. It's just when a second version has failed to 
start up, and we deploy a third version, only the first version is shut 
down, and the second version still appears to be running, with 0% traffic. 
The behavior of the gcloud app deploy command in this case is misleading.

On Saturday, March 18, 2017 at 12:39:42 PM UTC-4, Adam (Cloud Platform 
Support) wrote:
>
> Are you using manual scaling or automatic scaling in your app.yaml? If 
> you're using manual scaling, App Engine will try to keep instances alive 
> indefinitely 
> <https://cloud.google.com/appengine/docs/flexible/nodejs/how-instances-are-managed#instance_uptime>
>  
> until you manually shut them down.
>
> On Thursday, March 16, 2017 at 3:10:07 PM UTC-4, Alan deLespinasse wrote:
>>
>> I'm not sure if this is a bug or expected behavior. It's not what I 
>> expected, but I'm still somewhat confused by the Cloud Console.
>>
>> I have several services for which I noticed there were still some old 
>> versions running, although the old versions were serving 0% of traffic. 
>> Example:
>>
>>
>> <https://lh3.googleusercontent.com/-YZWAxJMW7x4/WMrgCXinq6I/Mac/iRAXXz_vhjYU9072hPy7ZXGENs4bvoCfQCLcB/s1600/oldservices.png>
>>
>> The two selected versions were made obsolete by the top version. Normally 
>> when I deploy a new version, the previously running version is shut down. 
>> In these cases, I think a new version never succeeded in starting up, 
>> because I had a bug that would simply crash every time. So here's what 
>> apparently happens:
>>
>>1. Version A is running
>>2. I try to deploy version B; it repeatedly fails to start up. My 
>>"gcloud app deploy" command eventually returns an error, telling me that 
>>deployment failed. Version A is still serving all traffic. Version B is 
>>apparently still running, though I didn't know that until just now.
>>3. I fix the bug and deploy version C. It starts up, version A is 
>>correctly stopped. Apparently version B is still running, though.
>>
>> So, is this a bug or expected? Is it costing us money? I'm finding it a 
>> bit hard to tell what our recent charges are for, although they are a bit 
>> higher than expected (still looking into that). I'd submit a billing ticket 
>> if I was sure.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/674f64f8-a717-48ba-aa3b-49849a136595%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Old versions still running (flexible environment)

[Alan deLespinasse]

I'm not sure if this is a bug or expected behavior. It's not what I 
expected, but I'm still somewhat confused by the Cloud Console.

I have several services for which I noticed there were still some old 
versions running, although the old versions were serving 0% of traffic. 
Example:



The two selected versions were made obsolete by the top version. Normally 
when I deploy a new version, the previously running version is shut down. 
In these cases, I think a new version never succeeded in starting up, 
because I had a bug that would simply crash every time. So here's what 
apparently happens:

   1. Version A is running
   2. I try to deploy version B; it repeatedly fails to start up. My 
   "gcloud app deploy" command eventually returns an error, telling me that 
   deployment failed. Version A is still serving all traffic. Version B is 
   apparently still running, though I didn't know that until just now.
   3. I fix the bug and deploy version C. It starts up, version A is 
   correctly stopped. Apparently version B is still running, though.

So, is this a bug or expected? Is it costing us money? I'm finding it a bit 
hard to tell what our recent charges are for, although they are a bit 
higher than expected (still looking into that). I'd submit a billing ticket 
if I was sure.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/e0baec81-8ed9-43d6-8660-e8dfa62a5d74%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Unable to deploy

[Alan deLespinasse]

Me too, and appengine.google.com isn't even working... it says Server 
Error. And the email address that it shows in the upper right isn't mine; 
it looks like an internal Google service account. I can only assume that 
something is seriously wrong, and some poor Google engineer who probably 
wasn't even awake yet has now been paged and is currently in panic mode.

The previous version of my app is still running, though.

On Tuesday, March 4, 2014 11:16:34 AM UTC-5, Nikolas Gauvreau wrote:

 I have the same problem now. 

 On Monday, 3 June 2013 02:46:10 UTC-4, stal...@hipergeo.net wrote:

 Was an answer ever forthcoming? I am having the same problem.

 On Monday, February 18, 2013 3:07:38 PM UTC-8, John Conti wrote:

 I am unable to upload my app:

 bash-3.2$ /usr/local/lib/appengine-java-sdk-1.7.5/bin/appcfg.sh update 
 war
 Reading application configuration data...
 Feb 18, 2013 4:00:52 PM 
 com.google.apphosting.utils.config.AppEngineWebXmlReader readAppEngineWebXml
 INFO: Successfully processed war/WEB-INF/appengine-web.xml
 Feb 18, 2013 4:00:52 PM 
 com.google.apphosting.utils.config.AbstractConfigXmlReader readConfigXml
 INFO: Successfully processed war/WEB-INF/web.xml
 Beginning server interaction for john-conti...
 0% Created staging directory at: 
 '/var/folders/_f/5xjr_n1n00d9n9z9ffwylp5mgn/T/appcfg344253979033234870.tmp'
 5% Using java7 runtime: false
 8% Scanning for jsp files.
 20% Scanning files on local disk.
 25% Initiating update.

 com.google.appengine.tools.admin.HttpIoException: Error posting to URL: 
 https://appengine.google.com/api/appversion/create?app_id=john-contiversion=1;
 404 Not Found
 This application does not exist (app_id=u'john-conti').

 Unable to update app: Error posting to URL: 
 https://appengine.google.com/api/appversion/create?app_id=john-contiversion=1;
 404 Not Found
 This application does not exist (app_id=u'john-conti').

 Please see the logs 
 [/var/folders/_f/5xjr_n1n00d9n9z9ffwylp5mgn/T/appcfg6215812047732459518.log]
  
 for further information.
 bash-3.2$ 


 The logs claim my app id doesn't exist.  However it is the same I have 
 been using all along, and my Dashboard confirms its existence and that it 
 is running properly:

 Unable to update:
 com.google.appengine.tools.admin.HttpIoException: Error posting to URL: 
 https://appengine.google.com/api/appversion/create?app_id=john-contiversion=1;
 404 Not Found
 This application does not exist (app_id=u'john-conti').

 at 
 com.google.appengine.tools.admin.AbstractServerConnection.send1(AbstractServerConnection.java:293)
 at 
 com.google.appengine.tools.admin.AbstractServerConnection.send(AbstractServerConnection.java:253)
 at 
 com.google.appengine.tools.admin.AbstractServerConnection.post(AbstractServerConnection.java:232)
 at 
 com.google.appengine.tools.admin.AppVersionUpload.send(AppVersionUpload.java:644)
 at 
 com.google.appengine.tools.admin.AppVersionUpload.beginTransaction(AppVersionUpload.java:449)
 at 
 com.google.appengine.tools.admin.AppVersionUpload.doUpload(AppVersionUpload.java:124)
 at 
 com.google.appengine.tools.admin.AppAdminImpl.doUpdate(AppAdminImpl.java:371)
 at 
 com.google.appengine.tools.admin.AppAdminImpl.update(AppAdminImpl.java:53)
 at 
 com.google.appengine.tools.admin.AppCfg$UpdateAction.execute(AppCfg.java:1163)
 at com.google.appengine.tools.admin.AppCfg.init(AppCfg.java:232)
 at com.google.appengine.tools.admin.AppCfg.init(AppCfg.java:109)
 at com.google.appengine.tools.admin.AppCfg.main(AppCfg.java:105)
 com.google.appengine.tools.admin.AdminException: Unable to update app: 
 Error posting to URL: 
 https://appengine.google.com/api/appversion/create?app_id=john-contiversion=1;
 404 Not Found
 This application does not exist (app_id=u'john-conti').

 at 
 com.google.appengine.tools.admin.AppAdminImpl.doUpdate(AppAdminImpl.java:376)
 at 
 com.google.appengine.tools.admin.AppAdminImpl.update(AppAdminImpl.java:53)
 at 
 com.google.appengine.tools.admin.AppCfg$UpdateAction.execute(AppCfg.java:1163)
 at com.google.appengine.tools.admin.AppCfg.init(AppCfg.java:232)
 at com.google.appengine.tools.admin.AppCfg.init(AppCfg.java:109)
 at com.google.appengine.tools.admin.AppCfg.main(AppCfg.java:105)
 Caused by: com.google.appengine.tools.admin.HttpIoException: Error 
 posting to URL: 
 https://appengine.google.com/api/appversion/create?app_id=john-contiversion=1;
 404 Not Found
 This application does not exist (app_id=u'john-conti').

 at 
 com.google.appengine.tools.admin.AbstractServerConnection.send1(AbstractServerConnection.java:293)
 at 
 com.google.appengine.tools.admin.AbstractServerConnection.send(AbstractServerConnection.java:253)
 at 
 com.google.appengine.tools.admin.AbstractServerConnection.post(AbstractServerConnection.java:232)
 at 
 com.google.appengine.tools.admin.AppVersionUpload.send(AppVersionUpload.java:644)
 at 
 com.google.appengine.tools.admin.AppVersionUpload.beginTransaction(AppVersionUpload.java:449)
 at 
 

[google-appengine] Re: Unable to deploy

[Alan deLespinasse]

Is the admin console broken for everyone? It's been completely broken for 
me all day.

On Tuesday, March 4, 2014 2:00:54 PM UTC-5, Nikolas Gauvreau wrote:

 If someone could post here if they manage to get deployments working that 
 would be helpful! That way I'll know if its an individual problem or an 
 appengine-being-down issue.

 On Tuesday, 4 March 2014 11:51:17 UTC-5, Chris wrote:

 Same problem people I was scared!!!but viewing this I hope that this just 
 was error...

 El martes, 19 de febrero de 2013 00:07:38 UTC+1, John Conti escribió:

 I am unable to upload my app:

 bash-3.2$ /usr/local/lib/appengine-java-sdk-1.7.5/bin/appcfg.sh update 
 war
 Reading application configuration data...
 Feb 18, 2013 4:00:52 PM 
 com.google.apphosting.utils.config.AppEngineWebXmlReader readAppEngineWebXml
 INFO: Successfully processed war/WEB-INF/appengine-web.xml
 Feb 18, 2013 4:00:52 PM 
 com.google.apphosting.utils.config.AbstractConfigXmlReader readConfigXml
 INFO: Successfully processed war/WEB-INF/web.xml
 Beginning server interaction for john-conti...
 0% Created staging directory at: 
 '/var/folders/_f/5xjr_n1n00d9n9z9ffwylp5mgn/T/appcfg344253979033234870.tmp'
 5% Using java7 runtime: false
 8% Scanning for jsp files.
 20% Scanning files on local disk.
 25% Initiating update.

 com.google.appengine.tools.admin.HttpIoException: Error posting to URL: 
 https://appengine.google.com/api/appversion/create?app_id=john-contiversion=1;
 404 Not Found
 This application does not exist (app_id=u'john-conti').

 Unable to update app: Error posting to URL: 
 https://appengine.google.com/api/appversion/create?app_id=john-contiversion=1;
 404 Not Found
 This application does not exist (app_id=u'john-conti').

 Please see the logs 
 [/var/folders/_f/5xjr_n1n00d9n9z9ffwylp5mgn/T/appcfg6215812047732459518.log]
  
 for further information.
 bash-3.2$ 


 The logs claim my app id doesn't exist.  However it is the same I have 
 been using all along, and my Dashboard confirms its existence and that it 
 is running properly:

 Unable to update:
 com.google.appengine.tools.admin.HttpIoException: Error posting to URL: 
 https://appengine.google.com/api/appversion/create?app_id=john-contiversion=1;
 404 Not Found
 This application does not exist (app_id=u'john-conti').

 at 
 com.google.appengine.tools.admin.AbstractServerConnection.send1(AbstractServerConnection.java:293)
 at 
 com.google.appengine.tools.admin.AbstractServerConnection.send(AbstractServerConnection.java:253)
 at 
 com.google.appengine.tools.admin.AbstractServerConnection.post(AbstractServerConnection.java:232)
 at 
 com.google.appengine.tools.admin.AppVersionUpload.send(AppVersionUpload.java:644)
 at 
 com.google.appengine.tools.admin.AppVersionUpload.beginTransaction(AppVersionUpload.java:449)
 at 
 com.google.appengine.tools.admin.AppVersionUpload.doUpload(AppVersionUpload.java:124)
 at 
 com.google.appengine.tools.admin.AppAdminImpl.doUpdate(AppAdminImpl.java:371)
 at 
 com.google.appengine.tools.admin.AppAdminImpl.update(AppAdminImpl.java:53)
 at 
 com.google.appengine.tools.admin.AppCfg$UpdateAction.execute(AppCfg.java:1163)
 at com.google.appengine.tools.admin.AppCfg.init(AppCfg.java:232)
 at com.google.appengine.tools.admin.AppCfg.init(AppCfg.java:109)
 at com.google.appengine.tools.admin.AppCfg.main(AppCfg.java:105)
 com.google.appengine.tools.admin.AdminException: Unable to update app: 
 Error posting to URL: 
 https://appengine.google.com/api/appversion/create?app_id=john-contiversion=1;
 404 Not Found
 This application does not exist (app_id=u'john-conti').

 at 
 com.google.appengine.tools.admin.AppAdminImpl.doUpdate(AppAdminImpl.java:376)
 at 
 com.google.appengine.tools.admin.AppAdminImpl.update(AppAdminImpl.java:53)
 at 
 com.google.appengine.tools.admin.AppCfg$UpdateAction.execute(AppCfg.java:1163)
 at com.google.appengine.tools.admin.AppCfg.init(AppCfg.java:232)
 at com.google.appengine.tools.admin.AppCfg.init(AppCfg.java:109)
 at com.google.appengine.tools.admin.AppCfg.main(AppCfg.java:105)
 Caused by: com.google.appengine.tools.admin.HttpIoException: Error 
 posting to URL: 
 https://appengine.google.com/api/appversion/create?app_id=john-contiversion=1;
 404 Not Found
 This application does not exist (app_id=u'john-conti').

 at 
 com.google.appengine.tools.admin.AbstractServerConnection.send1(AbstractServerConnection.java:293)
 at 
 com.google.appengine.tools.admin.AbstractServerConnection.send(AbstractServerConnection.java:253)
 at 
 com.google.appengine.tools.admin.AbstractServerConnection.post(AbstractServerConnection.java:232)
 at 
 com.google.appengine.tools.admin.AppVersionUpload.send(AppVersionUpload.java:644)
 at 
 com.google.appengine.tools.admin.AppVersionUpload.beginTransaction(AppVersionUpload.java:449)
 at 
 com.google.appengine.tools.admin.AppVersionUpload.doUpload(AppVersionUpload.java:124)
 at 
 com.google.appengine.tools.admin.AppAdminImpl.doUpdate(AppAdminImpl.java:371)
 ... 5 more


 Any ideas? 

  



-- 
You 

[google-appengine] Re: Unable to deploy

[Alan deLespinasse]

Seems to be working again (both the admin console and deployment).

On Tuesday, March 4, 2014 6:51:14 PM UTC-5, Alan deLespinasse wrote:

 Is the admin console broken for everyone? It's been completely broken for 
 me all day.

 On Tuesday, March 4, 2014 2:00:54 PM UTC-5, Nikolas Gauvreau wrote:

 If someone could post here if they manage to get deployments working that 
 would be helpful! That way I'll know if its an individual problem or an 
 appengine-being-down issue.

 On Tuesday, 4 March 2014 11:51:17 UTC-5, Chris wrote:

 Same problem people I was scared!!!but viewing this I hope that this 
 just was error...

 El martes, 19 de febrero de 2013 00:07:38 UTC+1, John Conti escribió:

 I am unable to upload my app:

 bash-3.2$ /usr/local/lib/appengine-java-sdk-1.7.5/bin/appcfg.sh update 
 war
 Reading application configuration data...
 Feb 18, 2013 4:00:52 PM 
 com.google.apphosting.utils.config.AppEngineWebXmlReader 
 readAppEngineWebXml
 INFO: Successfully processed war/WEB-INF/appengine-web.xml
 Feb 18, 2013 4:00:52 PM 
 com.google.apphosting.utils.config.AbstractConfigXmlReader readConfigXml
 INFO: Successfully processed war/WEB-INF/web.xml
 Beginning server interaction for john-conti...
 0% Created staging directory at: 
 '/var/folders/_f/5xjr_n1n00d9n9z9ffwylp5mgn/T/appcfg344253979033234870.tmp'
 5% Using java7 runtime: false
 8% Scanning for jsp files.
 20% Scanning files on local disk.
 25% Initiating update.

 com.google.appengine.tools.admin.HttpIoException: Error posting to URL: 
 https://appengine.google.com/api/appversion/create?app_id=john-contiversion=1;
 404 Not Found
 This application does not exist (app_id=u'john-conti').

 Unable to update app: Error posting to URL: 
 https://appengine.google.com/api/appversion/create?app_id=john-contiversion=1;
 404 Not Found
 This application does not exist (app_id=u'john-conti').

 Please see the logs 
 [/var/folders/_f/5xjr_n1n00d9n9z9ffwylp5mgn/T/appcfg6215812047732459518.log]
  
 for further information.
 bash-3.2$ 


 The logs claim my app id doesn't exist.  However it is the same I have 
 been using all along, and my Dashboard confirms its existence and that it 
 is running properly:

 Unable to update:
 com.google.appengine.tools.admin.HttpIoException: Error posting to URL: 
 https://appengine.google.com/api/appversion/create?app_id=john-contiversion=1;
 404 Not Found
 This application does not exist (app_id=u'john-conti').

 at 
 com.google.appengine.tools.admin.AbstractServerConnection.send1(AbstractServerConnection.java:293)
 at 
 com.google.appengine.tools.admin.AbstractServerConnection.send(AbstractServerConnection.java:253)
 at 
 com.google.appengine.tools.admin.AbstractServerConnection.post(AbstractServerConnection.java:232)
 at 
 com.google.appengine.tools.admin.AppVersionUpload.send(AppVersionUpload.java:644)
 at 
 com.google.appengine.tools.admin.AppVersionUpload.beginTransaction(AppVersionUpload.java:449)
 at 
 com.google.appengine.tools.admin.AppVersionUpload.doUpload(AppVersionUpload.java:124)
 at 
 com.google.appengine.tools.admin.AppAdminImpl.doUpdate(AppAdminImpl.java:371)
 at 
 com.google.appengine.tools.admin.AppAdminImpl.update(AppAdminImpl.java:53)
 at 
 com.google.appengine.tools.admin.AppCfg$UpdateAction.execute(AppCfg.java:1163)
 at com.google.appengine.tools.admin.AppCfg.init(AppCfg.java:232)
 at com.google.appengine.tools.admin.AppCfg.init(AppCfg.java:109)
 at com.google.appengine.tools.admin.AppCfg.main(AppCfg.java:105)
 com.google.appengine.tools.admin.AdminException: Unable to update app: 
 Error posting to URL: 
 https://appengine.google.com/api/appversion/create?app_id=john-contiversion=1;
 404 Not Found
 This application does not exist (app_id=u'john-conti').

 at 
 com.google.appengine.tools.admin.AppAdminImpl.doUpdate(AppAdminImpl.java:376)
 at 
 com.google.appengine.tools.admin.AppAdminImpl.update(AppAdminImpl.java:53)
 at 
 com.google.appengine.tools.admin.AppCfg$UpdateAction.execute(AppCfg.java:1163)
 at com.google.appengine.tools.admin.AppCfg.init(AppCfg.java:232)
 at com.google.appengine.tools.admin.AppCfg.init(AppCfg.java:109)
 at com.google.appengine.tools.admin.AppCfg.main(AppCfg.java:105)
 Caused by: com.google.appengine.tools.admin.HttpIoException: Error 
 posting to URL: 
 https://appengine.google.com/api/appversion/create?app_id=john-contiversion=1;
 404 Not Found
 This application does not exist (app_id=u'john-conti').

 at 
 com.google.appengine.tools.admin.AbstractServerConnection.send1(AbstractServerConnection.java:293)
 at 
 com.google.appengine.tools.admin.AbstractServerConnection.send(AbstractServerConnection.java:253)
 at 
 com.google.appengine.tools.admin.AbstractServerConnection.post(AbstractServerConnection.java:232)
 at 
 com.google.appengine.tools.admin.AppVersionUpload.send(AppVersionUpload.java:644)
 at 
 com.google.appengine.tools.admin.AppVersionUpload.beginTransaction(AppVersionUpload.java:449)
 at 
 com.google.appengine.tools.admin.AppVersionUpload.doUpload

Re: [google-appengine] Channel API JS partially broken

[Alan deLespinasse]

I was just working on a problem that has the same symptoms, but happened 
consistently every time (on Chrome). The problem turned out to be that the 
Disconnect plugin was overzealously blocking any Google API stuff, 
including the talkgadget URL.

Just thought I'd mention it for anyone who has the same problem and finds 
this page.

On Tuesday, November 15, 2011 11:03:31 AM UTC-5, Andrin von Rechenberg 
wrote:

 The one time I had this problem in chrome and looked at the source,
 the file was completely blank. Unfortunately it's hard to reproduce.

 -Andrin

 On Thu, Nov 10, 2011 at 12:52 AM, Pascal Patry 
 ppa...@spectrumdt.comjavascript:
  wrote:

 On Wednesday, November 09, 2011 16:28:29 Andrin von Rechenberg wrote:
  Hey there
 
  /_ah/channel/jsapi returns an empty file for 0.3% of clients
 
  We heavily use the Channel API javascript library (we do more than
  1'000'000 calls a day).
 
  About every 300th session the file returned by /_ah/channel/jsapi is
  completely blank and
  therefore new goog.appengine.Channel(id); fails, because goog is not
  defined. We see
  this more often happening on Android Devices than desktop computers.
 
  The most annoying thing however is, that this blank js file is then 
 cached
  on the client side for
  a very long time and reloading the page doesnt help to fix the js error.
  You need to clear your cache to fix it.
 
  We have seen this issue consistently for a couple of weeks now and were
  able to measure it continuously.
 
  I have filed a production bug here:
  http://code.google.com/p/googleappengine/issues/detail?id=6294
  Please star it if you use the Channel API.

 I have been hunting down this issue as well. We often get it on iOS as
 well as Android.

 --
 You received this message because you are subscribed to the Google Groups 
 Google App Engine group.
 To post to this group, send email to 
 google-a...@googlegroups.comjavascript:
 .
 To unsubscribe from this group, send email to 
 google-appengi...@googlegroups.com javascript:.
 For more options, visit this group at 
 http://groups.google.com/group/google-appengine?hl=en.




-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Dev datastore disappears when upgrading to new version

[Alan deLespinasse]

I recently installed the SDK version 1.8 (was using 1.7.7), and my local 
datastore in the dev_appserver has been reset to empty. Is this expected? 
Is the old data lying around somewhere that I could restore it? It's not a 
big deal (I would have been backing up the data if it was important), but 
it would be handy to be able to access it.

Also it would be nice if the installer would give a warning that it's about 
to delete data. I don't even see it in the release notes (which I didn't 
read before installing), but a warning during the install seems like the 
best thing to do.

This happened once before, I think around version 1.7.6 when the new 
dev_appserver appeared. In fact I can get at my pre-1.7.6 data by running 
old_dev_appserver.py, but I don't see a way to access my 1.7.7 data.

Developing in Python 1.7 on Windows; using the App Engine Launcher with 
pretty much default settings.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.