As a follow up on this -- task queue throttles at queries per second, not dequeues per second. The current limit is 20 qps. You can't fool it by using long running request handlers. I tried it in production.
Still looking for ways to get more than 20 qps for background jobs. Has anyone successfully done this? I would consider using an external box to generate inbound traffic if that's the only way, although I'm reluctant to do that. -- James On Oct 28, 12:00 pm, James Cooper <jamespcoo...@gmail.com> wrote: > aha! I missed that. I wonder if "task invocations/second" means > "dequeues/second". > > If it means dequeues/second then in theory you could write a request > handler that burns through a queue of work items in the datastore, re- > queueing itself and exiting after 25 seconds and achieve 250 CPU > seconds/second of concurrency. > > Is that crazy talk? > > I hope these limits go up when Tasks Queues exits beta. Google is > selling us computer time but is setting some fairly low limits on what > we're allowed to buy. 10 cores of 1.2ghz CPU time is roughly > equivalent to a modern 4 core desktop machine right? > > -- James > > On Oct 28, 12:27 pm, Scott Hernandez <scotthernan...@gmail.com> wrote: > > > The docs here seem to indicate that dequeuing happens at 5/sec > > (default and 10 > > max).http://code.google.com/appengine/docs/java/taskqueue/overview.html > > > On Oct 27, 8:41 am, James Cooper <jamespcoo...@gmail.com> wrote: > > > > Hi there, > > > > Last night I experimented with task queues to see what level of > > > concurrency I could achieve when running on the live environment. > > > > Summary of the test app: > > > - Bulk load 30,000 entities of a given type (3 properties / entity > > > object). > > > - Command line job I ran from my PC that hit an URL to queue the > > > entries > > > - This program was multi-threaded so I could simulate a bit of > > > load (10 concurrent threads) > > > - Queueing URL created a task queue entry within the same app > > > - 2nd URL handled the task queue request and stored entity to the > > > Datastore > > > > I watched the task queue dashboard for a few minutes and observed a > > > few things: > > > - Enqueue rate quickly outpaced dequeue rate > > > - I was enqueing at about 12 requests / second, but dequeuing at > > > 4 requests / second > > > - GAE did not appear to increase the dequeue rate over time in > > > response to my queue depth > > > > Result: It took a very long time to dequeue 30,000 tasks (over 2 > > > hours). It seemed that GAE was running one instance of my app. > > > > Expected: Much higher throughput. > > > > Is this expected behavior? It seems that given the 30 second request > > > limit that task queues are an important way to increase throughput > > > (alaMapReduce). But the "swarm" of app instances never seemed to > > > arrive. > > > > thanks > > > > -- James -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=.
