Hi Nick / Google, During the last 1-2 weeks, I've had the same problem. I've *never* encountered a backlog on my queues (load is small and processing generally fast). Then, suddenly and out of the blue, with *no* application or other changes, I get the behavior displayed on the attached graph (graph from Stackdriver, queue latency, 1month graph). It really crippled me!
I don't believe that this issue is related to my project. It must be a generic issue that you should be aware of. I've even tried with different AppEngine instance classes, but this "delay" had nothing to do with the instance "power". I don't think opening tickets will do any good. I've done it in the past, when I reported an issue with task queue rate (Issue #12620, acknowledged since December 17, 2015). Resolution -> NONE. I really love the AppEngine concept and development environment. But I doubt Google's devotion to it and that makes me really sad. On Friday, October 7, 2016 at 1:45:15 AM UTC+3, Nick (Cloud Platform Support) wrote: > > Hey Pasha, > > I would suggest posting a more detailed technical report to the Public > Issue Tracker <http://code.google.com/p/googleappengine/issues/list>. The > Public Issue Tracker is much better suited to issues like this, while this > forum generally should be geared toward high-level discussion of the > platform and services, design patterns, etc. > > You should try to include some or all of the following: > > 1. your project ID > 2. the task queue names > 3. their configurations > 4. the code for and/or a description of the task handler > 5. the time-frames when the issue is observed to occur within the past few > days (since time is of the essence in making sure the logs still exist in > the system) > 6. a description of the average load and the frequency distribution of > different sized loads > 7. the frequency distribution of response latencies for the task handlers > > This will enable us to get started on reasoning about the issue and > looking into it. Feel free to post a link to the thread here once you've > done that. I'll be happy to assist. > > Cheers, > > Nick > Cloud Platform Community Support > > On Thursday, October 6, 2016 at 5:26:56 PM UTC-4, Pasha Seliverstov wrote: >> >> Hi Nick, >> I have the same issue with queue not executing tasks at rate set in >> configuration. It looks really strange because the execution works >> correctly most of the days, but once per week the queue does not execute at >> correct rate. I experience the issue on Sep 30th and today Oct 6. >> >> I tried different configuration settings, but the configuration changes >> did affect the real execution rate. The execution is way too slow than set >> in configuration. >> >> Best regards, >> Pasha >> >> On Friday, May 13, 2016 at 11:44:57 AM UTC-7, Nick (Cloud Platform >> Support) wrote: >>> >>> Thanks for the update. Barring a very low setting for "max concurrent >>> tasks", I don't see why this could occur, so it appears this could be a >>> production issue. In that case, I'd recommend opening a Public Issue >>> Tracker <http://code.google.com/p/googleappengine/issues/list> thread >>> with label "Restrict-View-EditIssue" (to protect your information) and the >>> following information: >>> >>> * your app id >>> * the name of the queue >>> * the queue settings >>> * some queue statistics >>> >>> We may be able to determine the root cause of this at that point. >>> >>> Sincerely, >>> >>> Nick >>> Cloud Platform Community Support >>> >>> On Friday, May 13, 2016 at 2:02:04 PM UTC-4, Hugo Visser wrote: >>>> >>>> Just got an alert again. The task rate is 25 tasks/sec with a bucket >>>> size of 10. Average latency of the handlers is varying depending on the >>>> task, but anywhere between 1 and 8 seconds. >>>> >>>> The thing that I'm seeing is that the task queue shows 120 tasks in the >>>> queue, 0 running, oldest task ETA 6 minutes ago. That doesn't seem right >>>> to >>>> me, and it looks like these tasks are just not being executed. I see >>>> nothing in the logs of my task handlers that indicate that a request is >>>> being executed. >>>> >>>> Obviously this task execution thing is pretty essential to my app. >>>> >>>> Hugo >>>> >>>> On Wednesday, May 11, 2016 at 11:34:34 PM UTC+2, Nick (Cloud Platform >>>> Support) wrote: >>>>> >>>>> Thanks for coming back to update. Feel free to post again with the >>>>> information mentioned if you notice anything in the future. >>>>> >>>>> Best wishes, >>>>> >>>>> Nick >>>>> Cloud Platform Community Support >>>>> >>>>> On Wednesday, May 11, 2016 at 4:33:36 PM UTC-4, Hugo Visser wrote: >>>>>> >>>>>> It looks like the queues are back to normal. I've setup alerts in >>>>>> stack driver that triggered for a while but are resolved now. Will >>>>>> report >>>>>> back if that changes. >>>>>> >>>>>> On Wed, May 11, 2016, 21:08 'Nick (Cloud Platform Support)' via >>>>>> Google App Engine <google-a...@googlegroups.com> wrote: >>>>>> >>>>>>> Hey Hugo, >>>>>>> >>>>>>> Without knowing more about A) the configuration of the queue, B) the >>>>>>> rate of task creation, C) the latency statistics on the task handler, >>>>>>> it's >>>>>>> difficult to say much on why this might be happening. If you could >>>>>>> gather >>>>>>> all those types of information, myself or someone else may be able to >>>>>>> leave >>>>>>> a more insightful comment. >>>>>>> >>>>>>> Cheers, >>>>>>> >>>>>>> Nick >>>>>>> Cloud Platform Community Support >>>>>>> >>>>>>> >>>>>>> On Tuesday, May 10, 2016 at 4:54:00 AM UTC-4, Hugo Visser wrote: >>>>>>>> >>>>>>>> I'm seeing tasks in my queue that are there for 6 minutes without >>>>>>>> any tasks running at all. There doesn't seem to be much I can do about >>>>>>>> this >>>>>>>> which is really frustrating. >>>>>>>> >>>>>>>> On Tuesday, May 10, 2016 at 8:54:33 AM UTC+2, Hugo Visser wrote: >>>>>>>>> >>>>>>>>> Since last night or so I'm seeing this issue where tasks are being >>>>>>>>> queued up for some time and not being executed on an app that has >>>>>>>>> been >>>>>>>>> running like this for quite a while. This results in a huge pile of >>>>>>>>> tasks >>>>>>>>> that get backlogged for my app. >>>>>>>>> >>>>>>>>> Cloud status dashboard says everything is OK. Anyone else seeing >>>>>>>>> this? >>>>>>>>> >>>>>>>>> Hugo >>>>>>>>> >>>>>>>> -- >>>>>>> You received this message because you are subscribed to a topic in >>>>>>> the Google Groups "Google App Engine" group. >>>>>>> To unsubscribe from this topic, visit >>>>>>> https://groups.google.com/d/topic/google-appengine/MYxDL8sihN4/unsubscribe >>>>>>> . >>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>> google-appengi...@googlegroups.com. >>>>>>> To post to this group, send email to google-a...@googlegroups.com. >>>>>>> Visit this group at https://groups.google.com/group/google-appengine >>>>>>> . >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/google-appengine/370b31f3-e1fa-4a2f-9287-652aeac0500c%40googlegroups.com >>>>>>> >>>>>>> <https://groups.google.com/d/msgid/google-appengine/370b31f3-e1fa-4a2f-9287-652aeac0500c%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/7e996937-d775-47ae-9ba2-ebc177999c64%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
