[google-appengine] Re: Taskqueue execution latency up by 1000x

[Dan]

Looking at the logs I was a factor of 10 out. It is more like a 100x 
latency increase, not 1000x as taskqueue execution latency is usually < 
100ms, not 10ms . 1000x is much better for a post title though. ;-)

On Sunday, November 11, 2012 10:43:06 AM UTC, Dan wrote:
>
> Hi Goggle,
>
> Is it possible to have an update as to what's going on with the taskqueue 
> execution latency. According to the status monitor it has been going on 
> since midnight on 10th November and is still continuing.
>
> There are no notes to say you are looking into it so I just wanted to make 
> sure you are aware of it.
>
> The pattern I am seeing is that multiple tasks are delayed for over a 
> minute and then execute all at once. My app is used to an execution latency 
> in the order of < 10ms but I am seeing delays of over 60,000ms.
>
> Best wishes,
> Dan
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/KOX2gcvjSY4J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Taskqueue execution latency up by 1000x

[Marcel Manz]

Can someone from Google please respond to this issue?

The execution latency issue exists now for more than 48 hours and still no 
word or alerts about it. Can Google please allocate more human resources in 
order to communicate better? This is a business oriented service and such 
lack of communication must just not happen. And we don't care if such 
issues happen on weekends, App Engine needs to be stable 24/7.

- Marcel

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/GP59rYa-8LEJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Taskqueue execution latency up by 1000x

[Dan]

The system seems to be returning to normal after 60 hours of 60,000 - 
120,000 ms latency in taskqueue execution instead of 60 - 120 ms.

I am surprised users were not more vocal about this and no acknowledgement 
/ explanation from Google. Maybe people don't use taskqueues the way I use 
them - process some data on a taskqueue and then push it back to the UI. 
For this period of increased latency my users were getting their 
asynchronous data back in the UI one to two minutes after they had 
submitted it instead of the usual less than two seconds.

I imagine App Engine uses taskqueues as a way of smoothing out server 
traffic as they can stack them when load is high and run them when it is 
low and for the last 60 hours load has been continually high...

On Monday, November 12, 2012 12:00:19 PM UTC, Marcel Manz wrote:
>
> Can someone from Google please respond to this issue?
>
> The execution latency issue exists now for more than 48 hours and still no 
> word or alerts about it. Can Google please allocate more human resources in 
> order to communicate better? This is a business oriented service and such 
> lack of communication must just not happen. And we don't care if such 
> issues happen on weekends, App Engine needs to be stable 24/7.
>
> - Marcel
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/zlYWQaOKJVEJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Taskqueue execution latency up by 1000x

[stevep]

This issue has been raised before (I believe this is what you are seeing). 
Many moons ago vlad was its champion in these groups. His conclusion (link 
below): "I have been screaming about this for a looong time. The offcial 
position of GAE team is - we don't guarantee on time execution. Even though 
this issue is not about on time execution but rather about 
bugs/inefficiency of their scheduler." 

Sadly I think this is never going away, so best to build your client with 
the anticipation of such delays -- TQ tasks as part of handling client 
calls are too powerful/necessary to avoid entirely. HTH -stevep

Link:
https://groups.google.com/forum/?fromgroups=#!topic/google-appengine/nwnq7cjABhY



On Sunday, November 11, 2012 2:43:06 AM UTC-8, Dan wrote:
>
> Hi Goggle,
>
> Is it possible to have an update as to what's going on with the taskqueue 
> execution latency. According to the status monitor it has been going on 
> since midnight on 10th November and is still continuing.
>
> There are no notes to say you are looking into it so I just wanted to make 
> sure you are aware of it.
>
> The pattern I am seeing is that multiple tasks are delayed for over a 
> minute and then execute all at once. My app is used to an execution latency 
> in the order of < 10ms but I am seeing delays of over 60,000ms.
>
> Best wishes,
> Dan
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/Zwgucw2xah8J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Taskqueue execution latency up by 1000x

[Marcel Manz]

That issue seems related to task delays occuring for tasks that get 
scheduled from within tasks.

I assume that the taskqueue execution latency chart as per the status page 
does not relate to that issue and happened to be a generic task execution 
delay issue during the last days.



-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/UFbYkxq5nxMJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Taskqueue execution latency up by 1000x

[Dan]

Thanks for the info.

Looks like I spoke too soon as well, taskqueue execution latency is back up 
again - awesome. If only the latency increase was in the order of 10x not a 
massive 100x - 1000x...

On Tuesday, November 13, 2012 1:40:58 PM UTC, stevep wrote:
>
> This issue has been raised before (I believe this is what you are seeing). 
> Many moons ago vlad was its champion in these groups. His conclusion (link 
> below): "I have been screaming about this for a looong time. The offcial 
> position of GAE team is - we don't guarantee on time execution. Even though 
> this issue is not about on time execution but rather about 
> bugs/inefficiency of their scheduler." 
>
> Sadly I think this is never going away, so best to build your client with 
> the anticipation of such delays -- TQ tasks as part of handling client 
> calls are too powerful/necessary to avoid entirely. HTH -stevep
>
> Link:
>
> https://groups.google.com/forum/?fromgroups=#!topic/google-appengine/nwnq7cjABhY
>
>
>
> On Sunday, November 11, 2012 2:43:06 AM UTC-8, Dan wrote:
>>
>> Hi Goggle,
>>
>> Is it possible to have an update as to what's going on with the taskqueue 
>> execution latency. According to the status monitor it has been going on 
>> since midnight on 10th November and is still continuing.
>>
>> There are no notes to say you are looking into it so I just wanted to 
>> make sure you are aware of it.
>>
>> The pattern I am seeing is that multiple tasks are delayed for over a 
>> minute and then execute all at once. My app is used to an execution latency 
>> in the order of < 10ms but I am seeing delays of over 60,000ms.
>>
>> Best wishes,
>> Dan
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/fBCqJR-2_KMJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Taskqueue execution latency up by 1000x

[vlad]

I ditched TaskQueue for all requests with timeouts of less than 60 sec. It 
was simply too unreliable. Btw, there is a ticket filed for feature 
requesting a TaskQueue with timing guarantees. Feel free to star it. Sorry, 
too lazy to dig out a link.


On Tuesday, November 13, 2012 8:57:47 AM UTC-8, Dan wrote:
>
> Thanks for the info.
>
> Looks like I spoke too soon as well, taskqueue execution latency is back 
> up again - awesome. If only the latency increase was in the order of 10x 
> not a massive 100x - 1000x...
>
> On Tuesday, November 13, 2012 1:40:58 PM UTC, stevep wrote:
>>
>> This issue has been raised before (I believe this is what you are 
>> seeing). Many moons ago vlad was its champion in these groups. His 
>> conclusion (link below): "I have been screaming about this for a looong 
>> time. The offcial position of GAE team is - we don't guarantee on time 
>> execution. Even though this issue is not about on time execution but rather 
>> about bugs/inefficiency of their scheduler." 
>>
>> Sadly I think this is never going away, so best to build your client with 
>> the anticipation of such delays -- TQ tasks as part of handling client 
>> calls are too powerful/necessary to avoid entirely. HTH -stevep
>>
>> Link:
>>
>> https://groups.google.com/forum/?fromgroups=#!topic/google-appengine/nwnq7cjABhY
>>
>>
>>
>> On Sunday, November 11, 2012 2:43:06 AM UTC-8, Dan wrote:
>>>
>>> Hi Goggle,
>>>
>>> Is it possible to have an update as to what's going on with the 
>>> taskqueue execution latency. According to the status monitor it has been 
>>> going on since midnight on 10th November and is still continuing.
>>>
>>> There are no notes to say you are looking into it so I just wanted to 
>>> make sure you are aware of it.
>>>
>>> The pattern I am seeing is that multiple tasks are delayed for over a 
>>> minute and then execute all at once. My app is used to an execution latency 
>>> in the order of < 10ms but I am seeing delays of over 60,000ms.
>>>
>>> Best wishes,
>>> Dan
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/rxIPu2eId4wJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Taskqueue execution latency up by 1000x

[Doug Anderson]

Have you tried increasing the bucket_size above the default value of 5 
and/or the processing rate above 5/s?  I'm not saying there isn't a bug... 
I'm just curious if there's a workaround or if perhaps the problem is 
triggered by exceeding some threshold from which task queue manager never 
properly recovers.

On Tuesday, November 13, 2012 11:57:47 AM UTC-5, Dan wrote:
>
> Thanks for the info.
>
> Looks like I spoke too soon as well, taskqueue execution latency is back 
> up again - awesome. If only the latency increase was in the order of 10x 
> not a massive 100x - 1000x...
>
> On Tuesday, November 13, 2012 1:40:58 PM UTC, stevep wrote:
>>
>> This issue has been raised before (I believe this is what you are 
>> seeing). Many moons ago vlad was its champion in these groups. His 
>> conclusion (link below): "I have been screaming about this for a looong 
>> time. The offcial position of GAE team is - we don't guarantee on time 
>> execution. Even though this issue is not about on time execution but rather 
>> about bugs/inefficiency of their scheduler." 
>>
>> Sadly I think this is never going away, so best to build your client with 
>> the anticipation of such delays -- TQ tasks as part of handling client 
>> calls are too powerful/necessary to avoid entirely. HTH -stevep
>>
>> Link:
>>
>> https://groups.google.com/forum/?fromgroups=#!topic/google-appengine/nwnq7cjABhY
>>
>>
>>
>> On Sunday, November 11, 2012 2:43:06 AM UTC-8, Dan wrote:
>>>
>>> Hi Goggle,
>>>
>>> Is it possible to have an update as to what's going on with the 
>>> taskqueue execution latency. According to the status monitor it has been 
>>> going on since midnight on 10th November and is still continuing.
>>>
>>> There are no notes to say you are looking into it so I just wanted to 
>>> make sure you are aware of it.
>>>
>>> The pattern I am seeing is that multiple tasks are delayed for over a 
>>> minute and then execute all at once. My app is used to an execution latency 
>>> in the order of < 10ms but I am seeing delays of over 60,000ms.
>>>
>>> Best wishes,
>>> Dan
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/-EI5yWqYQmMJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Taskqueue execution latency up by 1000x

[Saurabh]

Hi Google,

The TaskQueue is experiencing HUGE latency in execution (as of 15:00 EST): 
it is totally messing up our mobile app: usually, TQ has been rock-solid 
reliable in chewing through the tasks, but it is really slow to execute 
things today.

Please address / acknowledge this since it is really wreaking havoc with 
the async apps that needed to get a lot of data processed for users!
Best regards,
Saurabh


On Sunday, November 11, 2012 5:43:06 AM UTC-5, Dan wrote:
>
> Hi Goggle,
>
> Is it possible to have an update as to what's going on with the taskqueue 
> execution latency. According to the status monitor it has been going on 
> since midnight on 10th November and is still continuing.
>
> There are no notes to say you are looking into it so I just wanted to make 
> sure you are aware of it.
>
> The pattern I am seeing is that multiple tasks are delayed for over a 
> minute and then execute all at once. My app is used to an execution latency 
> in the order of < 10ms but I am seeing delays of over 60,000ms.
>
> Best wishes,
> Dan
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/QXxszdo3lB8J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Taskqueue execution latency up by 1000x

[stevep]

I do not think the origination point of a TQ task (on-line handler, 
deferred function, or other TQ task) will affect its scheduling. There is a 
tiny bit of info about the TQ scheduler (it differs from "The Instance 
Scheduler"). Task scheduling is not always intuitive nor easily understood 
was my docs takeaway. HTH -stevep

On Tuesday, November 13, 2012 7:41:29 AM UTC-8, Marcel Manz wrote:
>
> That issue seems related to task delays occuring for tasks that get 
> scheduled from within tasks.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/ROXecCkZo60J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Taskqueue execution latency up by 1000x

[stevep]

Yet an architecture with such potential to do good. Someone call the 
Justice League!!! :-) -stevep

On Tuesday, November 13, 2012 11:24:48 AM UTC-8, vlad wrote:
>
> I ditched TaskQueue for all requests with timeouts of less than 60 sec. It 
> was simply too unreliable. Btw, there is a ticket filed for feature 
> requesting a TaskQueue with timing guarantees. Feel free to star it. Sorry, 
> too lazy to dig out a link.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/MecjmtFwWv8J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Taskqueue execution latency up by 1000x

[Saurabh Gupta]

Latency went down for a bit but has risen again:
http://code.google.com/status/appengine/detail/taskqueue/2012/11/13#ae-trust-detail-taskqueue-execution-latency

I've moved stuff over to execute on a backend: hope that can hold up until 
TQ latency becomes reasonable again.

Is this latency related to the nexus launch? Isn't the TQ designed to 
mitigate exactly this situation?

{SG}


On Sunday, November 11, 2012 5:43:06 AM UTC-5, Dan wrote:
>
> Hi Goggle,
>
> Is it possible to have an update as to what's going on with the taskqueue 
> execution latency. According to the status monitor it has been going on 
> since midnight on 10th November and is still continuing.
>
> There are no notes to say you are looking into it so I just wanted to make 
> sure you are aware of it.
>
> The pattern I am seeing is that multiple tasks are delayed for over a 
> minute and then execute all at once. My app is used to an execution latency 
> in the order of < 10ms but I am seeing delays of over 60,000ms.
>
> Best wishes,
> Dan
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/VsNUMpXmYWkJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Taskqueue execution latency up by 1000x

[Julie Smith]

FYI, it is issue #4901 at
http://code.google.com/p/googleappengine/issues/detail?id=4901


On 14 November 2012 06:24, vlad  wrote:

> I ditched TaskQueue for all requests with timeouts of less than 60 sec. It
> was simply too unreliable. Btw, there is a ticket filed for feature
> requesting a TaskQueue with timing guarantees. Feel free to star it. Sorry,
> too lazy to dig out a link.
>
>
>
> On Tuesday, November 13, 2012 8:57:47 AM UTC-8, Dan wrote:
>>
>> Thanks for the info.
>>
>> Looks like I spoke too soon as well, taskqueue execution latency is back
>> up again - awesome. If only the latency increase was in the order of 10x
>> not a massive 100x - 1000x...
>>
>> On Tuesday, November 13, 2012 1:40:58 PM UTC, stevep wrote:
>>>
>>> This issue has been raised before (I believe this is what you are
>>> seeing). Many moons ago vlad was its champion in these groups. His
>>> conclusion (link below): "I have been screaming about this for a looong
>>> time. The offcial position of GAE team is - we don't guarantee on time
>>> execution. Even though this issue is not about on time execution but rather
>>> about bugs/inefficiency of their scheduler."
>>>
>>> Sadly I think this is never going away, so best to build your client
>>> with the anticipation of such delays -- TQ tasks as part of handling client
>>> calls are too powerful/necessary to avoid entirely. HTH -stevep
>>>
>>> Link:
>>> https://groups.google.com/**forum/?fromgroups=#!topic/**
>>> google-appengine/nwnq7cjABhY
>>>
>>>
>>>
>>> On Sunday, November 11, 2012 2:43:06 AM UTC-8, Dan wrote:

 Hi Goggle,

 Is it possible to have an update as to what's going on with the
 taskqueue execution latency. According to the status monitor it has been
 going on since midnight on 10th November and is still continuing.

 There are no notes to say you are looking into it so I just wanted to
 make sure you are aware of it.

 The pattern I am seeing is that multiple tasks are delayed for over a
 minute and then execute all at once. My app is used to an execution latency
 in the order of < 10ms but I am seeing delays of over 60,000ms.

 Best wishes,
 Dan

>>>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/rxIPu2eId4wJ.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Taskqueue execution latency up by 1000x

[Saurabh Gupta]

Thanks, Julie! That reference is really helpful!


On Tuesday, November 13, 2012 6:49:28 PM UTC-5, Julie wrote:
>
> FYI, it is issue #4901 at
> http://code.google.com/p/googleappengine/issues/detail?id=4901
>
>
> On 14 November 2012 06:24, vlad >wrote:
>
>> I ditched TaskQueue for all requests with timeouts of less than 60 sec. 
>> It was simply too unreliable. Btw, there is a ticket filed for feature 
>> requesting a TaskQueue with timing guarantees. Feel free to star it. Sorry, 
>> too lazy to dig out a link.
>>
>>
>>
>> On Tuesday, November 13, 2012 8:57:47 AM UTC-8, Dan wrote:
>>>
>>> Thanks for the info.
>>>
>>> Looks like I spoke too soon as well, taskqueue execution latency is back 
>>> up again - awesome. If only the latency increase was in the order of 10x 
>>> not a massive 100x - 1000x...
>>>
>>> On Tuesday, November 13, 2012 1:40:58 PM UTC, stevep wrote:

 This issue has been raised before (I believe this is what you are 
 seeing). Many moons ago vlad was its champion in these groups. His 
 conclusion (link below): "I have been screaming about this for a looong 
 time. The offcial position of GAE team is - we don't guarantee on time 
 execution. Even though this issue is not about on time execution but 
 rather 
 about bugs/inefficiency of their scheduler." 

 Sadly I think this is never going away, so best to build your client 
 with the anticipation of such delays -- TQ tasks as part of handling 
 client 
 calls are too powerful/necessary to avoid entirely. HTH -stevep

 Link:
 https://groups.google.com/**forum/?fromgroups=#!topic/**
 google-appengine/nwnq7cjABhY



 On Sunday, November 11, 2012 2:43:06 AM UTC-8, Dan wrote:
>
> Hi Goggle,
>
> Is it possible to have an update as to what's going on with the 
> taskqueue execution latency. According to the status monitor it has been 
> going on since midnight on 10th November and is still continuing.
>
> There are no notes to say you are looking into it so I just wanted to 
> make sure you are aware of it.
>
> The pattern I am seeing is that multiple tasks are delayed for over a 
> minute and then execute all at once. My app is used to an execution 
> latency 
> in the order of < 10ms but I am seeing delays of over 60,000ms.
>
> Best wishes,
> Dan
>
  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Google App Engine" group.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msg/google-appengine/-/rxIPu2eId4wJ.
>>
>> To post to this group, send email to 
>> google-a...@googlegroups.com
>> .
>> To unsubscribe from this group, send email to 
>> google-appengi...@googlegroups.com .
>> For more options, visit this group at 
>> http://groups.google.com/group/google-appengine?hl=en.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/Rcff7nNPQO4J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.