If your external system is called synchronously by your tasks, you could try setting the queue's max_concurrent_requests parameter in queue.yaml/xml
This is documented for Java, but the same general rules apply for Python or Go applications. http://code.google.com/appengine/docs/java/config/queue.html#Defining_Push_Queues_and_Processing_Rates Nick Verne On Mon, Oct 31, 2011 at 3:13 PM, Pol <p...@everpix.net> wrote: > Hi, > > Through a task queue, we manage calls from GAE to an external system, > which is fully-scalable but takes minutes to do so. The task queue > rate is therefore set to 200/s. > > So what happens is that sometimes we get bursts of activity, and the > task queue sends 200 requests at once: the vast majority fails in the > external system as it can't handle this sudden request in load. But > the external system doesn't start scaling up as there are no more > requests coming and it's scaling based on the CPU load. Then suddenly > you get another burst of requests from the task queue retrying, and > then again the majority fails and so on. > > So how can we configure a task queue to have a very high rate *but* > prevent it to get to this rate too fast? I noticed the bucket size > parameter, but I'm not sure how to use it properly. > > Thanks, > > - Pol > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
