[google-appengine] Re: another wrinkle with GAE/Python Managed VMs: timezone set to local deploy location, not UTC like in GAE

[Michael Sander]

Yes, datetime.utcnow() does work correctly, however, many libraries that 
are out of my direct control call datetime.now().  This is especially 
apparent when using django, which uses datetime.now() all throughout its 
code-base. For example, the auto_now feature in django automatically sets 
the time of the database item you are saving to be datetime.now(), which 
can result in items in your database being set incorrectly. Further, the 
popular humanize django plugin (which humanizes dates and time deltas) may 
also behave differently from expected.

This is blocking using managed VM with the very popular django web 
framework.

On Thursday, March 17, 2016 at 5:55:24 PM UTC-4, Adam Sah wrote:
>
> ah... utcnow() works correctly and indeed that's the ideal solution for 
> apps that can update their code.
>
> thanks!
> adam
>
> On Thursday, March 17, 2016 at 2:09:42 PM UTC-7, Nick (Cloud Platform 
> Support) wrote:
>>
>> Hey Folks,
>>
>> Here's the PIT thread to track this 
>> <https://code.google.com/p/googleappengine/issues/detail?id=12840=12840=1458248968>.
>>  
>> Expect any updates to be posted there.
>>
>> On Thursday, March 17, 2016 at 4:40:45 PM UTC-4, Jon Parrott wrote:
>>>
>>> To further Nick's point here, Python provides datetime.datetime.utcnow() 
>>> which allows you to explicitly get the current time in UTC. It's almost 
>>> always preferable to use that unless you explicitly want to deal with 
>>> system-local time.
>>>
>>> On Wednesday, March 16, 2016 at 12:30:33 PM UTC-7, Nick (Cloud Platform 
>>> Support) wrote:
>>>>
>>>> Hey Michael,
>>>>
>>>> We're currently working on getting this resolved, although the best way 
>>>> to track its resolution is by posting and following a Public Issue Tracker 
>>>> issue. At any rate, we're currently working on it and it should be 
>>>> resolved 
>>>> soon. If your application is sensitive to the system time-zone, the 
>>>> natural 
>>>> recourse is to set it explicitly, rather than assume a default. 
>>>> Nonetheless, UTC is a sensible default I believe and should be 
>>>> implemented. 
>>>>
>>>> I'll update this thread with any progress on the issue.
>>>>
>>>> Best wishes,
>>>>
>>>> Nick
>>>> Cloud Platform Community Support
>>>>
>>>> On Wednesday, March 16, 2016 at 2:01:46 PM UTC-4, Michael Sander wrote:
>>>>>
>>>>> I've seen this too, but I just started working on Managed VM. Has this 
>>>>> been around for a while? It seems like a pretty major bug for it not to 
>>>>> be 
>>>>> fixed. Also, the workaround is totally nuts. Is this going to be fixed or 
>>>>> is the hack our only recourse?
>>>>>
>>>>> On Thursday, March 10, 2016 at 6:54:04 PM UTC-5, Nick (Cloud Platform 
>>>>> Support) wrote:
>>>>>>
>>>>>> Hey Adam,
>>>>>>
>>>>>> Glad to see you found a workaround, I was going to some lengths to 
>>>>>> verbally describe what you basically ended up doing. You should post a 
>>>>>> quick description of this issue to the Public Issue Tracker for App 
>>>>>> Engine <https://code.google.com/p/googleappengine/issues/list> and 
>>>>>> we'll be happy to take a look. It seems rational to have UTC as the 
>>>>>> default.
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Nick
>>>>>> Cloud Platform Community Support
>>>>>>
>>>>>> On Thursday, March 10, 2016 at 6:28:39 PM UTC-5, Adam Sah wrote:
>>>>>>>
>>>>>>> OK, horrible workaround deployed.  Roughly:
>>>>>>>
>>>>>>> main.py:
>>>>>>>
>>>>>>> PATHS=[...
>>>>>>>   (r'/_ah/warmup', WarmupView),
>>>>>>>   (r'/_ah/start', StartView),
>>>>>>>
>>>>>>> class WarmupView(webapp.RequestHandler):
>>>>>>>   def get(self, *args):  # pylint:disable=W0613,R0201
>>>>>>> base.SERVER_TZ = None  # allow base.set_now() to reset server 
>>>>>>> timezone
>>>>>>> base.now()
>>>>>>>
>>>>>>> class StartView(webapp.RequestHandler):
>>>>>>>   def get(self, *args)

[google-appengine] Re: another wrinkle with GAE/Python Managed VMs: timezone set to local deploy location, not UTC like in GAE

[Michael Sander]

I've seen this too, but I just started working on Managed VM. Has this been 
around for a while? It seems like a pretty major bug for it not to be 
fixed. Also, the workaround is totally nuts. Is this going to be fixed or 
is the hack our only recourse?

On Thursday, March 10, 2016 at 6:54:04 PM UTC-5, Nick (Cloud Platform 
Support) wrote:
>
> Hey Adam,
>
> Glad to see you found a workaround, I was going to some lengths to 
> verbally describe what you basically ended up doing. You should post a 
> quick description of this issue to the Public Issue Tracker for App Engine 
>  and we'll be 
> happy to take a look. It seems rational to have UTC as the default.
>
> Cheers,
>
> Nick
> Cloud Platform Community Support
>
> On Thursday, March 10, 2016 at 6:28:39 PM UTC-5, Adam Sah wrote:
>>
>> OK, horrible workaround deployed.  Roughly:
>>
>> main.py:
>>
>> PATHS=[...
>>   (r'/_ah/warmup', WarmupView),
>>   (r'/_ah/start', StartView),
>>
>> class WarmupView(webapp.RequestHandler):
>>   def get(self, *args):  # pylint:disable=W0613,R0201
>> base.SERVER_TZ = None  # allow base.set_now() to reset server timezone
>> base.now()
>>
>> class StartView(webapp.RequestHandler):
>>   def get(self, *args):  # pylint:disable=W0613,R0201
>> base.SERVER_TZ = None  # allow base.set_now() to reset server timezone
>> base.now()
>>
>> base.py:
>>
>> import pytz
>> ...
>> SERVER_TZ = None
>> ...
>> def now(nowval=None):
>>   global SERVER_TZ
>>   ...
>>   if SERVER_TZ is None:
>>   try:
>> # HACK of the year award!  os.environ['TZ'] isn't available 
>> during GAE/Managed VMs startup
>> # and date +%Z and /etc/timezone didn't have useful information 
>> either
>> # even env | grep TZ failed!!
>> import subprocess
>> localtz = subprocess.check_output("date +%Z", shell=True).strip()
>> # SUPER HACK: I couldn't find pytz-compatible timezone strings, 
>> so I hacked it.
>> # Yes, I know I'm going to hell for this.
>> SERVER_TZ = 'America/Chicago' if localtz == 'CST' else 'UTC'
>>   except:
>> SERVER_TZ = 'UTC'
>> nowval = datetime.datetime.now()  # pylint:disable=W9914
>> if SERVER_TZ != 'UTC':
>>   # set the tz to the server tz, convert to UTC, then strip the tz 
>> info
>>   nowval = nowval.replace(tzinfo=pytz.timezone(SERVER_TZ)).astimezone(
>> pytz.timezone('UTC')).replace(tzinfo=None)
>>   return nowval
>>
>>
>> adam
>>
>>
>> On Thursday, March 10, 2016 at 1:01:57 PM UTC-8, Adam Sah wrote:
>>>
>>> oops, to be specific here's what I'm seeing:
>>>
>>>datetime.datetime.now().strftime("%c %Z") ==> Thu Mar 10 14:57:31 
>>> 2016(i.e. timestamp is in CST with no timezone info)
>>>os.environ['TZ'] ==> America/Chicago
>>>
>>> where dev_appserver and standard GAE return:
>>>
>>>datetime.datetime.now().strftime("%c %Z") ==> Thu Mar 10 21:00:14 
>>> 2016(i.e. timestamp is in UTC with no timezone info)
>>>os.environ['TZ'] ==> 'UTC'
>>>
>>> adam
>>>
>>> On Thursday, March 10, 2016 at 12:32:07 PM UTC-8, Adam Sah wrote:

 Here's what I've tried so far...
  - set os.environ['TZ'] to 'UTC' on startup - failed because GAE 
 overwrites this.
  - set os.environ['TZ'] to 'UTC' on each call by creating a child class 
 to webapp.WSGIApplication and adding it to __call__() - failed because GAE 
 overwrites this.
  - monkeypatching datetime.datetime.now() to use a specially hacked 
 function - failed because GAE disallows modifying now()

 I'm currently trying a hack to rewrite all calls (100+) 
 to datetime.datetime.now() to instead use an app-specific implementation 
 which returns the UTC time - this won't work for third party libraries, 
 but 
 that maybe OK for my app, which uses pylint custom rules to migrate 
 engineers to using app-specific replacement functions (e.g. we disallow 
 direct calls to gc.collect() so we can track stats better).  If this 
 works, 
 I'll post.

 adam


 On Wednesday, March 9, 2016 at 1:45:54 PM UTC-8, Adam Sah wrote:
>
> e.g. os.environ['TZ'] changed to America/Chicago when before it was 
> UTC.
>
> The docs don't mention this difference anywhere.
>
> If anyone has a workaround, pls reply - I'm working on a workaround 
> now, will post if successful.
>
> hope this helps,
> adam
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 

[google-appengine] Re: Old versions of Managed VMs

[Michael Sander]

Just ran into this issue today, and noticed that as of 3/9/2016 it was 
fixed. Time to do a "gcloud components update"

On Monday, October 5, 2015 at 12:39:29 PM UTC-4, Michael Spainhower wrote:
>
> Ah, found the answer myself in the gcloud release notes at 
> https://dl.google.com/dl/cloudsdk/release/RELEASE_NOTES
>
> In a future Cloud SDK release, deployments that promote the new version
>>   to receive all traffic will stop the previous version by default.
>>
>>
> Great to hear.  Thanks! 
>
> On Sunday, October 4, 2015 at 4:02:29 PM UTC-4, Michael Spainhower wrote:
>>
>> Could anyone from Google comment on whether there is a plan to fix this 
>> issue?  Versions that are not getting traffic really shouldn't have 
>> instances constantly running.  Only using (and paying for) the resources 
>> that you really need and use is one of the key value props of Google Cloud 
>> and GAE.
>>
>> On Tuesday, September 15, 2015 at 7:15:10 PM UTC-4, Jeff Schnitzer wrote:
>>>
>>> I have a python app on a Managed VM which I deploy with:
>>>
>>> gcloud preview app deploy app.yaml --remote --set-default
>>>
>>> It's set to manual scaling, instances 1.
>>>
>>> It appears that every time I deploy it, the old instance stick around 
>>> (and get billed for). Even deleting the old versions from the appengine 
>>> console didn't shut down the compute engine instances. I had to figure out 
>>> which compute engine instance was my "live" one and delete all the others 
>>> from the Compute Engine part of the console.
>>>
>>> Is this supposed to work this way or is there a better workflow? I just 
>>> want to deploy the new version and have the old version go away... like 
>>> regular App Engine.
>>>
>>> Jeff
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/2be9a93c-4762-415b-833e-d690053d2ee5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Taskqueue Failing with Error Code 123 Before 10 Minute deadline

[Michael Sander]

Hi Christian - Good thought. I took a look at the code in C:\Program Files 
(x86)\Google\google_appengine\google\appengine\ext\deferred and nothing 
jumped out at me as being different or special. The only major added 
complexity is the introduction of pickle in the deferred library. But this 
does not seem like a pickle issue.

On Wednesday, January 20, 2016 at 2:19:12 PM UTC-5, Christian F. Howes 
wrote:
>
> is there any chance that deferred tasks are somehow treated differently 
> then regularly queued tasks?  in theory i would think they are just tasks 
> with a special handler, but could that be why you are hitting an incorrect 
> deadline?
>
> On Wednesday, January 20, 2016 at 7:27:01 AM UTC-8, Nicholas (Google Cloud 
> Support) wrote:
>>
>> Thank you for posting your error logs. According to 'Google Cloud Status 
>> <https://status.cloud.google.com/>', there are no know outages for Task 
>> Queues or App Engine modules at the moment.
>>
>> As you've mentioned, the default deadline for a automatic scaled modules 
>> is 10 minutes. This is documented in 'Task deadlines 
>> <https://cloud.google.com/appengine/docs/python/taskqueue/overview-push#task_deadlines>'.
>>  
>> Please note that the deadline can be as high as 24 hours for manual or 
>> basic scaling modules. Also note that the DeadlineExceededException is 
>> raised when the task's execution time nears the deadline, thus allowing you 
>> to save your work or log something before the deadline is actually 
>> exceeded. That being said, more information is required to get a better 
>> idea of what might really be the culprit here:
>>
>>1. What is the .yaml configuration for the related task-handling 
>>modules? (feel free to censor any project-sensitive data)
>>2. What is the time frame? (When did this start; is it ongoing; if 
>>not, when did it end)
>>3. Have you encountered any other timeout or deadline exceeded errors 
>>from other modules, including those unrelated to task queues?
>>4. How is your queue.yaml configured 
>><https://cloud.google.com/appengine/docs/python/config/queue> for the 
>>queues experiencing these issues?
>>5. Do retries 
>>
>> <https://cloud.google.com/appengine/docs/python/taskqueue/overview-push#task_retries>
>>  or 
>>increased module scalability help reduce the frequency of these errors?
>>
>> Though the thread 'Suddenly App Engine not Serving Anymore: Error Code 
>> 123 
>> <https://groups.google.com/forum/#!msg/google-appengine/ujF7fzOmEew/0HbZFIehAAAJ>'
>>  
>> seems to describe similar errors, the scope of these errors and what 
>> systems are causing them is unclear in said thread so I would advise to 
>> continue here going forward.
>>
>> On Tuesday, January 19, 2016 at 9:01:51 PM UTC-5, Michael Sander wrote:
>>>
>>> Hi All,
>>>
>>> Over the past few days I've been getting a lot of Error Code 123 in my 
>>> Taksqueue. I understand that this error is a deadline, but Taskqueues are 
>>> supposed to live for 10 minutes, and I get this error far before that. It 
>>> is difficult to debug because these requests have practically no logging 
>>> information in them. My app normally spits out a lot of logs, so either GAE 
>>> is dropping my logs or this crash occurs outside of my code.
>>>
>>>
>>>1. 2016-01-19 20:36:45.991 /_ah/queue/deferred 500 190547ms 0kb 
>>> AppEngine-Google; 
>>>(+http://code.google.com/appengine) module=default version=18
>>>
>>>0.1.0.2 - - [19/Jan/2016:17:36:45 -0800] "POST /_ah/queue/deferred 
>>> HTTP/1.1" 500 0 "http://docketupdate.appspot.com/_ah/queue/deferred; 
>>> "AppEngine-Google; (+http://code.google.com/appengine)" 
>>> "docketupdate.appspot.com" ms=190547 cpu_ms=0 cpm_usd=0.65 
>>> queue_name=default task_name=37491651647795200701 exit_code=123 
>>> app_engine_release=1.9.31 instance=00c61b117c4864f5a842a8ae7ca496d20dbaa9f8 
>>> <https://appengine.google.com/instances?app_id=s~docketupdate_id=18.390103371601440022=00c61b117c4864f5a842a8ae7ca496d20dbaa9f8#00c61b117c4864f5a842a8ae7ca496d20dbaa9f8>
>>>
>>>2. E2016-01-19 20:36:45.991
>>>
>>>Process terminated because the request deadline was exceeded. (Error 
>>> code 123)
>>>
>>>
>>>
>>> Here's another:
>>>
>>>
>>>
>>>1. 2016-01-19 19:48:58.581 /_ah/queue/deferred 500 84488ms

[google-appengine] Re: Taskqueue Failing with Error Code 123 Before 10 Minute deadline

[Michael Sander]

Hi Nicholas,

Appreciate you jumping in. I purchased a Silver support package and that 
team has said they are on it (but no answer so far).

This is happening in my default module, here is the relevant section from 
app.yaml:

runtime: python27
threadsafe: yes
api_version: 1

instance_class: F4
automatic_scaling:
  min_idle_instances: 2
  max_idle_instances: 15
  max_pending_latency: 150ms

includes:
- libs/mapreduce/include.yaml

builtins:
- remote_api: on
- admin_redirect: on

inbound_services:
- warmup
- mail_bounce
- mail


2. I started noticing this on Jan. 18 at 7:30PM ET. At that time, it was a 
very big problem, I was getting thousands of these errors and my sight was 
unresponsive. Since then, I have (1) lowered the rate of my taskqueues, (2) 
lowered the rate of cron jobs, (3) increased min_idle_instances from 1 to 
2. The problem has decreased, but has not gone away.

3. I'm not sure if this is specific to Taskqueues. Below are two examples 
of a user requests failing with the same error message. Notice that the 
request time is 100 and 87 seconds respectively, so these would have hit 
the timeout. However, what is suspicious is that there are no logs printed 
other than the error message. Both of these requests should print out quite 
a bit of logging information.

   1. 2016-01-20 13:47:52.869 /get_creditcard.ajax?_=1453315570813 500 
   100831ms 0kb Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, 
   like Gecko) Chrome/47.0.2526.111 Safari/537.36 module=default version=18
   
   209.155.152.66 - - [20/Jan/2016:10:47:52 -0800] "GET 
/get_creditcard.ajax?_=1453315570813 HTTP/1.1" 500 0 - "Mozilla/5.0 (Windows NT 
6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.111 Safari/537.36" 
"www.docketalarm.com" ms=100832 cpu_ms=0 cpm_usd=0.74 exit_code=123 
app_engine_release=1.9.31 instance=00c61b117ca4e745b9e7703d6d0cad31c835274c 
<https://appengine.google.com/instances?app_id=s~docketupdate_id=18.390121211429959750=00c61b117ca4e745b9e7703d6d0cad31c835274c#00c61b117ca4e745b9e7703d6d0cad31c835274c>
   
   2. E2016-01-20 13:47:52.869
   
   Process terminated because the request deadline was exceeded. (Error code 
123)
   
   

   1. 2016-01-20 13:03:02.143 
   
/cases/TTAB/91119978/KELLOGG_COMPANY_v._PACIFIC_GRAIN_PRODUCTS_INC./docs/16.pdf?download=true
500 87681ms 0kb Mozilla/5.0 (Windows NT 5.1; rv:6.0.2) Gecko/20100101 
   Firefox/6.0.2 module=default version=18
   2. E2016-01-20 13:03:02.143 Process terminated because the request 
   deadline was exceeded. (Error code 123)


4. The issues mostly arise on queues that are running relatively quickly (> 
1.0/sec)

5. Yes, increasing min_idle_instances helped, but did not eliminate the 
issue. See #2 above.

Thanks,

On Wednesday, January 20, 2016 at 10:27:01 AM UTC-5, Nicholas (Google Cloud 
Support) wrote:
>
> Thank you for posting your error logs. According to 'Google Cloud Status 
> <https://status.cloud.google.com/>', there are no know outages for Task 
> Queues or App Engine modules at the moment.
>
> As you've mentioned, the default deadline for a automatic scaled modules 
> is 10 minutes. This is documented in 'Task deadlines 
> <https://cloud.google.com/appengine/docs/python/taskqueue/overview-push#task_deadlines>'.
>  
> Please note that the deadline can be as high as 24 hours for manual or 
> basic scaling modules. Also note that the DeadlineExceededException is 
> raised when the task's execution time nears the deadline, thus allowing you 
> to save your work or log something before the deadline is actually 
> exceeded. That being said, more information is required to get a better 
> idea of what might really be the culprit here:
>
>1. What is the .yaml configuration for the related task-handling 
>modules? (feel free to censor any project-sensitive data)
>2. What is the time frame? (When did this start; is it ongoing; if 
>not, when did it end)
>3. Have you encountered any other timeout or deadline exceeded errors 
>from other modules, including those unrelated to task queues?
>4. How is your queue.yaml configured 
><https://cloud.google.com/appengine/docs/python/config/queue> for the 
>queues experiencing these issues?
>5. Do retries 
>
> <https://cloud.google.com/appengine/docs/python/taskqueue/overview-push#task_retries>
>  or 
>increased module scalability help reduce the frequency of these errors?
>
> Though the thread 'Suddenly App Engine not Serving Anymore: Error Code 123 
> <https://groups.google.com/forum/#!msg/google-appengine/ujF7fzOmEew/0HbZFIehAAAJ>'
>  
> seems to describe similar errors, the scope of these errors and what 
> systems are causing them is unclear in said thread so I would advise to 
> continue here going forward.
>
> On Tuesday, January 

[google-appengine] Re: Suddenly App Engine not Serving Anymore: Error code 123

[Michael Sander]

What do you mean slowing down the app? Lowering the taskqueue execution 
speed? Cron speed?

On Tuesday, January 19, 2016 at 9:03:55 PM UTC-5, Kaan Soral wrote:
>
> Yes, manually slowing down the app almost diminishes these
>
> But I still see 40-50 of them every now and then
>
> On Wednesday, January 20, 2016 at 3:53:17 AM UTC+2, Michael Sander wrote:
>>
>> I've randomly started seeing these too. Did you do anything that worked?
>>
>> On Thursday, June 18, 2015 at 7:50:52 AM UTC-4, Kaan Soral wrote:
>>>
>>> I also see some outages from my logs
>>>
>>> The worst part is that the SLA refunds are not automatic, I have too 
>>> many things on my plate and the SLA refund/request process is too criptic
>>>
>>> On Wednesday, June 17, 2015 at 8:01:36 PM UTC+3, Karl-Heinz Müller wrote:
>>>>
>>>> I have a few app engines running, serving different domain names using 
>>>> the same application code php. 
>>>>
>>>> Two of them suddenly stopped serving content. In the log file I see the 
>>>> following error message:
>>>>
>>>> I 12:56:46.855 This request caused a new process to be started for 
>>>> your application, and thus caused your application code to be loaded for 
>>>> the first time. This request may thus take longer and use more CPU than a 
>>>> typical request for your application.
>>>> E 12:56:46.855 Process terminated because the request deadline was 
>>>> exceeded. (Error code 123)
>>>>
>>>>
>>>> I have another app engine running the same application without any issues 
>>>> and accessing the same data storage unit. Wondering why two shut down and 
>>>> the other not. 
>>>>
>>>>
>>>> No changes have been done. The incident started today 12:00
>>>>
>>>>
>>>> Thanks
>>>>
>>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/b4fdb79c-c7b9-46c5-8342-9ff26b252843%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Taskqueue Failing with Error Code 123 Before 10 Minute deadline

[Michael Sander]

Hi All,

Over the past few days I've been getting a lot of Error Code 123 in my 
Taksqueue. I understand that this error is a deadline, but Taskqueues are 
supposed to live for 10 minutes, and I get this error far before that. It 
is difficult to debug because these requests have practically no logging 
information in them. My app normally spits out a lot of logs, so either GAE 
is dropping my logs or this crash occurs outside of my code.


   1. 2016-01-19 20:36:45.991 /_ah/queue/deferred 500 190547ms 0kb 
AppEngine-Google; 
   (+http://code.google.com/appengine) module=default version=18
   
   0.1.0.2 - - [19/Jan/2016:17:36:45 -0800] "POST /_ah/queue/deferred HTTP/1.1" 
500 0 "http://docketupdate.appspot.com/_ah/queue/deferred; "AppEngine-Google; 
(+http://code.google.com/appengine)" "docketupdate.appspot.com" ms=190547 
cpu_ms=0 cpm_usd=0.65 queue_name=default task_name=37491651647795200701 
exit_code=123 app_engine_release=1.9.31 
instance=00c61b117c4864f5a842a8ae7ca496d20dbaa9f8 

   
   2. E2016-01-19 20:36:45.991
   
   Process terminated because the request deadline was exceeded. (Error code 
123)
   
   

Here's another:



   1. 2016-01-19 19:48:58.581 /_ah/queue/deferred 500 84488ms 0kb 
AppEngine-Google; 
   (+http://code.google.com/appengine) module=default version=18
   
   0.1.0.2 - - [19/Jan/2016:16:48:58 -0800] "POST /_ah/queue/deferred HTTP/1.1" 
500 0 "http://docketupdate.appspot.com/_ah/queue/deferred; "AppEngine-Google; 
(+http://code.google.com/appengine)" "docketupdate.appspot.com" ms=84489 
cpu_ms=0 queue_name=indexsearch task_name=5718499252586827456 exit_code=123 
app_engine_release=1.9.31 instance=00c61b117cbebad1360673a661a18cf0e69c7cb0 

   
   2. E2016-01-19 19:48:58.580
   
   Process terminated because the request deadline was exceeded. (Error code 
123)
   
   


And another:


   1. 2016-01-19 19:48:58.583 /_ah/queue/deferred 500 105886ms 0kb 
AppEngine-Google; 
   (+http://code.google.com/appengine) module=default version=18
   
   0.1.0.2 - - [19/Jan/2016:16:48:58 -0800] "POST /_ah/queue/deferred HTTP/1.1" 
500 0 "http://docketupdate.appspot.com/_ah/queue/deferred; "AppEngine-Google; 
(+http://code.google.com/appengine)" "docketupdate.appspot.com" ms=105887 
cpu_ms=0 cpm_usd=0.009467 queue_name=officialdocumentqueue 
task_name=8648504684653226301 exit_code=123 app_engine_release=1.9.31 
instance=00c61b117cbebad1360673a661a18cf0e69c7cb0 

   
   2. E2016-01-19 19:48:58.583
   
   Process terminated because the request deadline was exceeded. (Error code 
123)
   
   

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/8577d539-6ac5-49be-9cfa-18df43a0c3a0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Suddenly App Engine not Serving Anymore: Error code 123

[Michael Sander]

I've randomly started seeing these too. Did you do anything that worked?

On Thursday, June 18, 2015 at 7:50:52 AM UTC-4, Kaan Soral wrote:
>
> I also see some outages from my logs
>
> The worst part is that the SLA refunds are not automatic, I have too many 
> things on my plate and the SLA refund/request process is too criptic
>
> On Wednesday, June 17, 2015 at 8:01:36 PM UTC+3, Karl-Heinz Müller wrote:
>>
>> I have a few app engines running, serving different domain names using 
>> the same application code php. 
>>
>> Two of them suddenly stopped serving content. In the log file I see the 
>> following error message:
>>
>> I 12:56:46.855 This request caused a new process to be started for your 
>> application, and thus caused your application code to be loaded for the 
>> first time. This request may thus take longer and use more CPU than a 
>> typical request for your application.
>> E 12:56:46.855 Process terminated because the request deadline was 
>> exceeded. (Error code 123)
>>
>>
>> I have another app engine running the same application without any issues 
>> and accessing the same data storage unit. Wondering why two shut down and 
>> the other not. 
>>
>>
>> No changes have been done. The incident started today 12:00
>>
>>
>> Thanks
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/2e480a42-cb30-4729-9796-17b0496ac354%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] SSL Errors on App Engine, Please Help ASAP

[Michael Sander]

Hi,

My entire App Engine site is down, and I think it's due to incorrectly 
configured SSL. My Google App Engine custom domain: 
https://www.docketalarm.com

However, when I curl:

curl -XGET https://www.docketalarm.com
curl: (35) Unknown SSL protocol error in connection to 
www.docketalarm.com:443

I get a similar issue when I just go to https://www.docketalarm.com

In Google Apps, it instructs me to set the CNAME 
to: ghs-svc-https-c278.ghs-ssl.googlehosted.com
I did that for the www cname in godaddy.com, but now the entire site is 
down. Any help?

https://lh3.googleusercontent.com/-J0ix4LoIyRk/Va3KjhQSpHI/AIQ/j31YSnYbDFE/s1600/godaddy.png

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/94e7b284-b423-4128-9858-7d4895c328fc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: SSL Errors on App Engine, Please Help ASAP

[Michael Sander]

Okay, I did nothing, and now it's working.

On Tuesday, July 21, 2015 at 12:29:13 AM UTC-4, Michael Sander wrote:

 Hi,

 My entire App Engine site is down, and I think it's due to incorrectly 
 configured SSL. My Google App Engine custom domain: 
 https://www.docketalarm.com

 However, when I curl:

 curl -XGET https://www.docketalarm.com
 curl: (35) Unknown SSL protocol error in connection to 
 www.docketalarm.com:443

 I get a similar issue when I just go to https://www.docketalarm.com

 In Google Apps, it instructs me to set the CNAME to: 
 ghs-svc-https-c278.ghs-ssl.googlehosted.com
 I did that for the www cname in godaddy.com, but now the entire site is 
 down. Any help?


 https://lh3.googleusercontent.com/-J0ix4LoIyRk/Va3KjhQSpHI/AIQ/j31YSnYbDFE/s1600/godaddy.png



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/c45e6c39-80c7-43bc-b202-f04bf6bb20a0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Memory Usage Doubled in the Past 20 Hours or So

[Michael Sander]

Did google upgrade or change something recently? About 20 hours ago my app 
started consuming about twice as much memory as before. I have not updated 
or changed my app at all in the past few days. See screenshot below.

https://lh6.googleusercontent.com/-RPylKvSjchA/VQyQrQiQKUI/Ccg/9pZJ54ZTFr0/s1600/2015-03-20%2BScreenshot.png

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/873c2e73-7087-43a3-9643-de3c4539f3b3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Memory Usage Doubled in the Past 20 Hours or So

[Michael Sander]

Totally uncorrelated. Traffic hasn't changed at all... actually it's 
decreased a bit. Summary screenshot below.

https://lh3.googleusercontent.com/-N0wvSx7XKY8/VQyTERNZCkI/Ccs/qlp7PyXsEBk/s1600/2015-03-20%2BScreenshot%2B2.png


On Friday, March 20, 2015 at 5:31:41 PM UTC-4, Alex Martelli wrote:

 How have various measures of _traffic_ to your app been changing over the 
 last couple days?  I.e, is that doubling of memory quite uncorrelated with 
 an increase in traffic?

 Alex

 On Fri, Mar 20, 2015 at 2:27 PM, Michael Sander michael...@gmail.com 
 javascript: wrote:

 Did google upgrade or change something recently? About 20 hours ago my 
 app started consuming about twice as much memory as before. I have not 
 updated or changed my app at all in the past few days. See screenshot below.


 https://lh6.googleusercontent.com/-RPylKvSjchA/VQyQrQiQKUI/Ccg/9pZJ54ZTFr0/s1600/2015-03-20%2BScreenshot.png

  -- 
 You received this message because you are subscribed to the Google Groups 
 Google App Engine group.
 To unsubscribe from this group and stop receiving emails from it, send an 
 email to google-appengi...@googlegroups.com javascript:.
 To post to this group, send email to google-a...@googlegroups.com 
 javascript:.
 Visit this group at http://groups.google.com/group/google-appengine.
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/google-appengine/873c2e73-7087-43a3-9643-de3c4539f3b3%40googlegroups.com
  
 https://groups.google.com/d/msgid/google-appengine/873c2e73-7087-43a3-9643-de3c4539f3b3%40googlegroups.com?utm_medium=emailutm_source=footer
 .
 For more options, visit https://groups.google.com/d/optout.




-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/5cca5db1-0a84-4dba-8ec1-852e2863ab63%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Memory Usage Doubled in the Past 20 Hours or So

[Michael Sander]

This seems to be getting even worse I'm hitting 5GB of memory even though 
my load is pretty tiny.

https://lh4.googleusercontent.com/-27TrkW2v_ns/VQyeMtF5-hI/Cc8/Oar2yQzLCCw/s1600/2015-03-20%2BScreenshot%2B3.png



On Friday, March 20, 2015 at 5:37:39 PM UTC-4, Michael Sander wrote:

 Totally uncorrelated. Traffic hasn't changed at all... actually it's 
 decreased a bit. Summary screenshot below.


 https://lh3.googleusercontent.com/-N0wvSx7XKY8/VQyTERNZCkI/Ccs/qlp7PyXsEBk/s1600/2015-03-20%2BScreenshot%2B2.png


 On Friday, March 20, 2015 at 5:31:41 PM UTC-4, Alex Martelli wrote:

 How have various measures of _traffic_ to your app been changing over the 
 last couple days?  I.e, is that doubling of memory quite uncorrelated with 
 an increase in traffic?

 Alex

 On Fri, Mar 20, 2015 at 2:27 PM, Michael Sander michael...@gmail.com 
 wrote:

 Did google upgrade or change something recently? About 20 hours ago my 
 app started consuming about twice as much memory as before. I have not 
 updated or changed my app at all in the past few days. See screenshot below.


 https://lh6.googleusercontent.com/-RPylKvSjchA/VQyQrQiQKUI/Ccg/9pZJ54ZTFr0/s1600/2015-03-20%2BScreenshot.png

  -- 
 You received this message because you are subscribed to the Google 
 Groups Google App Engine group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to google-appengi...@googlegroups.com.
 To post to this group, send email to google-a...@googlegroups.com.
 Visit this group at http://groups.google.com/group/google-appengine.
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/google-appengine/873c2e73-7087-43a3-9643-de3c4539f3b3%40googlegroups.com
  
 https://groups.google.com/d/msgid/google-appengine/873c2e73-7087-43a3-9643-de3c4539f3b3%40googlegroups.com?utm_medium=emailutm_source=footer
 .
 For more options, visit https://groups.google.com/d/optout.




-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/ddee00c4-73d0-4dfc-9b38-61afb6fdc465%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Release 1.8.2 issue - RuntimeError: internal error in regular expression engine

[Michael Sander]

This appears to be a Python bug that was introduced in Python version 
2.7.5. I am guessing that when google upgraded to 1.8.1, it also upgraded 
Python which broke this behavior. This is causing failures in my production 
code. Google -- please revert to 1.8.1 ASAP!

On Wednesday, July 10, 2013 1:32:43 AM UTC-4, Mitul Golakiya wrote:

 Google had automatically migrated My Apps GAE SDK version to 1.8.2 from 
 1.8.1...
 And now getting the same issue after migrating from 1.8.1 to 1.8.2...

 Is there any option, by which I can specify my app's GAE SDK version ???

 On Wednesday, July 10, 2013 1:03:39 AM UTC+5:30, Kinesh Patel wrote:


 Getting exceptions on the instances that have the new SDK (1.8.2) in 
 Python 2.7

 nightloop/mailbot/parser.py, line 23, in _strip_suffix
 result = re.findall(regex, body, re.S|re.I)
   File 
 /base/data/home/runtimes/python27/python27_dist/lib/python2.7/re.py, 
 line 177, in findall
 return _compile(pattern, flags).findall(string)
 RuntimeError: internal error in regular expression engine

 App is currently broken because of it.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] Do F1 F2 F4 Google App Engine frontend instances really cost more

[Michael Sander]

I've run some tests and F2 is certainly faster than F1, especially at CPU 
intensive things such as generating templates. Unfortunately, it also seems 
to be close to double the price.

On Tuesday, July 24, 2012 1:25:04 PM UTC-4, Douglas Pollock wrote:

 Yes, as it stands right now, you need to restart the instances.  You can 
 do this by manually restarting the instances, or by deploying a new version 
 of the app.



 On Thu, Jul 19, 2012 at 12:59 AM, Marc Hacker z...@tradeos.comjavascript:
  wrote:

 Thanks yes we found that stopping instances is necessary to activate 
 changes in the F1/F2/F4 settings.

 But we are still getting F4 at most 2-3x faster than F1 even for a pure 
 CPU benchmark (for loop)


 On Thursday, July 19, 2012 10:50:22 AM UTC+3, Francois Masurel wrote:

 Have you tried to shutdown your currently running instances?  It should 
 help.

  -- 
 You received this message because you are subscribed to the Google Groups 
 Google App Engine group.
 To view this discussion on the web visit 
 https://groups.google.com/d/msg/google-appengine/-/Heey-25_BqkJ.

 To post to this group, send email to 
 google-a...@googlegroups.comjavascript:
 .
 To unsubscribe from this group, send email to 
 google-appengi...@googlegroups.com javascript:.
 For more options, visit this group at 
 http://groups.google.com/group/google-appengine?hl=en.




-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/L73qvRp0QEEJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Dev Server on PyPy

[Michael Sander]

I know it's not an officially supported platform, but has anyone tried 
playing around with the App Engine dev server on PyPy? Its JIT should make 
the dev server much faster than when running on cpython. With a bit of 
tinkering, I can get it to start up, but I quickly get the following 
exception whenever I try request a page on the dev server:

File C:\Program Files 
(x86)\Google\google_appengine\google\appengine\tools\dev_appserver_import_hook.py,
 
line 1606, in FindModuleRestricted
raise import_error
ImportError: Access to module file denied: c:\PyPy19\lib_pypy\_functools.py


The _functools.py file exists on my machine and is readable.  Any idea why 
it's broken?

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/fIAcI0X-9DIJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.