I'm doing this too. Turn that task into one that takes parameters - a start and end index. You set a threshold of the biggest number of tasks that can be kicked off in one execution. Let's say it's 50. So, you're given 1 to 5000... So, you kick off a task that spawns off new instances of itself from 1-50, 51-100, 101-150, etc. This task will do no logic but kick off tasks.
If your range is less than that threshold, then you can just process them. This strategy takes a lot of work, but it's totally worth it, because you've just made your app scale better. You can now handle much more than 5,000 tasks - if your logic is clever enough, there's no limit to how big that number can be. I think I just keep dividing by 10 until I have less than 10, then process those locally. I'm excited about the 1.3.2 bulk add API - I hadn't heard of that before. On Mar 25, 10:45 am, Eugene Kuleshov <ekules...@gmail.com> wrote: > My application need to create bunch of tasks to do some data > processing. I've tried to prototype a small application that spawns > 5000 tasks from the process initiated by cron job, but it seem like I > am hitting some wall, because my test can't spawn more then 1000 tasks > and it is terminated by GAE runtime after 30 seconds. > > Is there any known performance limitations of the Task Queue Java > API or any best practices on how to spawn large number of tasks? > > Thanks > > Eugene -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
