You mention that "tasks get rescheduled for some reason".. what is the reason? Does this reason occur frequently?
Also, there is no way to evaluate how fast you can perform your modifications since you haven't shown the code that you are currently using. There may be several simple tweaks to your existing code that could make it much faster. On Mon, Nov 29, 2010 at 9:29 AM, dflorey <daniel.flo...@gmail.com> wrote: > Thanks for your response. I though that mapreduce will also sit on top > of task queue and will most likely give any speed improvements over my > approach? > I am seeing ~1500 tasks per minute getting executed. Will mapreduce > give higher numbers? > > Daniel > > On 29 Nov., 10:41, Peter Ondruska <peter.ondru...@gmail.com> wrote: > > I would you mapreduce for GAE, seehttp:// > code.google.com/p/appengine-mapreduce/. > > It has been integrated with latest SDK so no need to download, I use > > it with Python--just make sure to import > > google.appengine.ext.mapreduce. > > > > On 29 lis, 10:06, dflorey <daniel.flo...@gmail.com> wrote: > > > > > > > > > > > > > > > > > Hi, > > > I'm looking for the most effective way to update 50000 entities + one > > > of the child entities each. > > > Right now I'm using a task per transaction to be able to modify the > > > entity and the child entities inside a transaction to make the task > > > idempotent. > > > I'm using sharded counters to check when the operation is done. > > > Everything works fine, but it takes very long (=minutes to hours) to > > > perform the modifications. > > > I'm getting no concurrent modification exceptions etc. at all, but > > > tasks get rescheduled for some reason and wait for a long time before > > > getting executed depending on the number of retries. > > > > > Is there a way to speed things up? > > > I'm looking for a solution that will execute the update almost > > > immediately :-) > > > My tasks take less than 1000ms each and I can see ~30 instances in the > > > dashboard. > > > > > Thanks for any ideas, > > > > > Daniel > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appeng...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
