When I batch update entities, I have two tasks: The first, iterates through the entity keys and encodes an update task for each entity that needs to be updated. That approach will probably get you to 400k within the 10m run of the iteration task, especially if you submit the tasks in batches of 100 or so. The google guava Lists.partition() method is handy here.
I haven't needed to do this yet, but you can also take the approach of splitting up the work. This is fairly easy if you have simple ids without parent keys. You start with the lowest key and the highest key, then you use a splitter task that divides and conquers the address space, spawning new splitter tasks until you reach what you consider to be a sufficiently small space to spawn iterator tasks. Just be careful to use the task queue's name mechanism to prevent duplicate tasks from being encoded when some of these tasks inevitably retry. I don't quite know what you do when you have parent keys. Presumably the map/reduce library has solved that - it would be worth looking into. It basically uses the same divide-and-conquer approach. Jeff On Tue, Mar 4, 2014 at 11:16 AM, Kaan Soral <kaanso...@gmail.com> wrote: > I haven't noticed something as critical as the issue you mentioned, but I > did notice some fishy behavior around taskqueue.BulkAdd - especially when > excess amount of tasks involed as you mentioned > > I was forced to switch to my own implementation of defer and defer task > routines, as appengine only recently started using ndb.toplevel, as far as I > recall, defer just adds the task to a taskqueue, so basically we can > consider you are talking about taskqueue > > It might be a good idea to start a new discussion about this subject and > clearly define these limitations > > I started utilizing a governer async container and flush routines to .wait > when the async elements reach to a limit, it might be a good idea to apply > this to taskqueue's too > I'm manually batching taskqueue calls by 50's - 500's are a bit problematic, > but it's too much trouble for defer-like tasks, where there is no order > > (I didn't refine the above post, so it might be a bit confusing, out of > touch, not well-defined) > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to google-appengine+unsubscr...@googlegroups.com. > To post to this group, send email to google-appengine@googlegroups.com. > Visit this group at http://groups.google.com/group/google-appengine. > For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
