Re: [google-appengine] Re: Task rate limits?

[Attila-Mihaly Balazs]

Just a quick note (and I'm sure that you're already aware of this, but for 
the benefit of others reading this): when "some tasks ... have failed [to 
be scheduled]" the task queue uses an exponential backoff algorithm, so the 
failed tasks will be delayed. See the configuration options in queue.yaml 
(https://cloud.google.com/appengine/docs/standard/python/config/queueref) - 
especially the "retry_parameters" and their default values.

If you browse your queue in the cloud console, you can see how many (if 
any) times a task has been retried. You can also check this from within the 
task by looking at the "X-AppEngine-TaskRetryCount" header.

Cheers,
Attila

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/36df5504-bb13-4e20-8a7f-3b9fbfe4334b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: Task rate limits?

['Jordan (Cloud Platform Support)' via Google App Engine]

You are exactly correct in your interpretation.  

To clarify 'ETA', this is an option that can be specified when you add a 
task to a queue. If an ETA is not provided, it will be set to 'now' (the 
creation time). The ETA designates the absolute earliest time a task should 
run, forcing workers to wait at least the 'ETA' before leasing the task. 
This is not an actual up-to-date dynamic ETA, but an option configured by 
the requester. Hence why you would see "8 mins ago" if the task hasn't been 
leased by a worker for 8 minutes past the ETA.

If you are looking for more alerting or monitoring for task queues, you can 
specify the details of the information you would like to see present via a 
Feature 
Request . 

 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/538fd8ea-34af-4c5b-87c7-0b0517ecd8fb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: Task rate limits?

[Emlyn]

Thanks for that feedback Jordan, that's potentially really useful.

So maybe you are proposing that something like this is happening:

- I enqueue a massive amount of tasks very quickly (and those tasks enqueue
more tasks etc).
- Inevitably, there is less capacity available (instances) than necessary.
So some tasks get *individually* delayed (they fail to be handled, and go
into exponential backoff).
- AppEngine keeps spinning up instances until the contention goes away
- But, some tasks may have failed to be scheduled enough times in the
interim that they now appear "stuck" for a little while. What's actually
going on is that they've been "backed off", and have a timestamp in the
future when they will try to run.

So, it'll look like some tasks are getting stuck.

Is that right?

Further, and crucially to me, is this invisible via the console? ie: would
these tasks be sitting there and not running, but I can't see any evidence
of why? I'm asking this because this isn't, for instance, affecting the
task's ETA (it might say something like "8 mins ago").




On 16 June 2017 at 06:21, 'Jordan (Cloud Platform Support)' via Google App
Engine  wrote:

> The limits on Task Queue calls can be found on the Quota page
> , specifically the
> 'Queue execution rate' being of importance.
>
> Too many tasks being executed on a single queue will inevitably result in
> underlying contention. This leads to the slow down of task execution (aka
> exponential-backoff retry) while tasks wait for resources to handle them.
> It is recommended to instead shard your tasks across multiple queues to get
> around this limit. You can also tweak your queue.yaml
> 
> *'max_concurrent_requests'* setting to prevent any single queue from
> hitting this limit.
>
> Additionally, it is always good to check your logs to ensure that you
> didn't see any errors during the gap, as you could be hitting other quotas
> limits (which would require time to refill).
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/google-appengine/308b61be-732f-4b05-8fe2-
> 925ac46c0874%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Emlyn

https://medium.com/the-infinite-machine - A publication about Google App
Engine
sutllang.com - My language sUTL
https://plus.google.com/u/0/100281903174934656260 - Google+

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CAMp1VPBojegLPJqbXSKrCPcf9Ma5u1rtAxPko%3DC_17_Se2f_vQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Task rate limits?

['Jordan (Cloud Platform Support)' via Google App Engine]

The limits on Task Queue calls can be found on the Quota page 
, specifically the 
'Queue execution rate' being of importance. 

Too many tasks being executed on a single queue will inevitably result in 
underlying contention. This leads to the slow down of task execution (aka 
exponential-backoff retry) while tasks wait for resources to handle them. 
It is recommended to instead shard your tasks across multiple queues to get 
around this limit. You can also tweak your queue.yaml  

*'max_concurrent_requests'* setting to prevent any single queue from 
hitting this limit. 

Additionally, it is always good to check your logs to ensure that you 
didn't see any errors during the gap, as you could be hitting other quotas 
limits (which would require time to refill).

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/308b61be-732f-4b05-8fe2-925ac46c0874%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: Task rate limits?

[Emlyn]

I don't see any yellow tooltips.


A bit more info: I'm not sure, but I think it applies to certain tasks, not
the entire queue. So maybe under load some rare tasks get into a temporary
error/stuck state?

On 14 June 2017 at 21:25, Attila-Mihaly Balazs  wrote:

> There seems to be some kind of "safety throttling" with the queues.
> Whenever I hit it, there is a yellow warning sign next to the queue in the
> cloud console: https://console.cloud.google.com/appengine/taskqueues and
> the tooltip says something about the queue being throttled to avoid
> impacting other customers.
>
> So perhaps you could check your cloud console and see if there are any
> warnings whenever the queue seems to be throttled?
>
> Attila
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/google-appengine/a0cff223-8a7d-4de3-bb4e-
> 28db0d0ad17a%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Emlyn

https://medium.com/the-infinite-machine - A publication about Google App
Engine
sutllang.com - My language sUTL
https://plus.google.com/u/0/100281903174934656260 - Google+

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CAMp1VPC1T3vzEO2qwk%3DBpnWb41keW2JrV%3D-YdJgxmegXd86iUQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Task rate limits?

[Attila-Mihaly Balazs]

There seems to be some kind of "safety throttling" with the queues. 
Whenever I hit it, there is a yellow warning sign next to the queue in the 
cloud console: https://console.cloud.google.com/appengine/taskqueues and 
the tooltip says something about the queue being throttled to avoid 
impacting other customers.

So perhaps you could check your cloud console and see if there are any 
warnings whenever the queue seems to be throttled?

Attila

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/a0cff223-8a7d-4de3-bb4e-28db0d0ad17a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.