Hi Greg, When I set a rate of say 30/M, I basically always see all of my tasks run in the first second or two then the queue sits empty for a minute, then 30 run, etc.... So are you saying if I adjusted the bucket size down to, say, 1, then they would at least trickle out over 30 seconds?
Robert On Tue, Feb 1, 2011 at 16:29, Greg Darke <da...@google.com> wrote: > Hi HalcyonDays, > > In this case the system will add another token every 1/50th of a second. > > Note that the taskqueue system may batch up requests so that even > though you will receive a new token every 1/50th of a second it may > burst 2-3 (well, actually up to 5) tasks at a time. > > On 2 February 2011 07:35, HalcyonDays <jbpap...@gmail.com> wrote: >> Thanks for replying... >> >> So in my example of rate=50/s and bucket-size=5: >> When no tasks are in the queue, clearly the bucket fills up and has 5 >> tokens. >> If I add 100 tasks, the first 5 will execute immediately and take all >> 5 tokens... but now the bucket is empty. >> >> I have specified a rate of 50 tokens to be added to the bucket every >> second... but there are two ways that could happen: >> 1) 50 tokens are deposited every 1 second. >> 2) 1 token is deposited every 1/50th of a second. >> >> If the first is true, then by using a bucket of size 5, no more than 5 >> tasks will ever be able to execute in a second because tokens are only >> being added 50 at a time to a bucket that can only hold 5 tokens. >> If the second is true, 5 will burst initially, but then there will be >> a steady stream of 1 task every 50th of a second. >> >> From what you said in your post: "the token-adding resolution is the >> unit of the rate you set..." it sounds like the first is true... which >> would mean that the task queue is essentially ALWAYS bursting. I >> guess that's not a huge deal, but means that to truly execute 50 tasks >> a second, I have to bump my bucket-size to 50 as well, in which case >> the task queue essentially starts 50 simultaneous tasks instead of >> staging them across the entire second... the behavior I would prefer >> is the second... >> >> On Feb 1, 1:44 pm, Robert Kluin <robert.kl...@gmail.com> wrote: >>> Hi, >>> The rate is like an average upper bound; in other words, the >>> token-adding resolution is the unit of the rate you set. And, it is >>> 'bursty,' so if you set a rate of 50/M you might very well have 50 >>> tasks execute in the first second then the queue will not execute any >>> tasks for about 59 seconds. Setting a higher bucket_size will let the >>> queue 'burst' at a higher rate, I usually see that after a queue has >>> emptied. >>> >>> At least that's how I've seen the task-queue behave. >>> >>> Robert >>> >>> >>> >>> On Mon, Jan 31, 2011 at 14:57, HalcyonDays <jbpap...@gmail.com> wrote: >>> > So I've read through a couple of things about the Task Queue's Token- >>> > Bucket system, and I've taken a look at the wikipedia article to no >>> > avail, so forgive me if this question has been answered elsewhere: >>> >>> > What is the resolution of the clock that deposits tokens into the >>> > bucket for each queue? Is its minimum resolution "per second?" >>> > i.e. If I say, "I want my rate to be 10/s," are 10 tokens deposited >>> > into the bucket every second, or is one token added every 10th of a >>> > second? >>> >>> > It seems like if the resolution is only per second, the rate is >>> > limited not only by the user's specification, but also by the bucket >>> > size... >>> >>> > For example: If I have a specified rate of 50/s but a bucket size of 5 >>> > and the system deposits 50 tokens into the bucket every second, are >>> > the additional 45 just thrown away? Does my rate actually become 5/s? >>> >>> > I'm specifically using the Java app engine, if there's a difference >>> > between the Task Queues in Python/Java. >>> >>> > -- >>> > You received this message because you are subscribed to the Google Groups >>> > "Google App Engine" group. >>> > To post to this group, send email to google-appengine@googlegroups.com. >>> > To unsubscribe from this group, send email to >>> > google-appengine+unsubscr...@googlegroups.com. >>> > For more options, visit this group >>> > athttp://groups.google.com/group/google-appengine?hl=en. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to google-appengine@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> >> > > > > -- > Greg Darke > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
