This is exactly the sort of task the MapReduce was meant for. It should be really a lot easier than managing the partitioning, error recovery, etc yourself. Take a look at our new docs: https://developers.google.com/appengine/docs/java/dataprocessing/mapreduce_library hopefully they should make it less overwhelming.
On Thu, Feb 27, 2014 at 5:04 AM, de Witte <wd.dewi...@gmail.com> wrote: > Use a backend instance and keep it running until done. > > Or > > Use two tasks. One for retrieving 1000 keys at the time and a second one > to update the entities in a batch of 1000. > > Done it for 300.000 entities in less than a 20 mins. ~300 tasks > > Op vrijdag 7 februari 2014 22:43:33 UTC+1 schreef Keith Lea: > >> Hi everyone, >> >> I'm a long time App Engine user for my app's backend, but I'm really >> still a novice about the datastore. >> >> I'd like to add a new property (and index) for all entities of a certain >> type. I have about 400,000 of this type of entity in the datastore, and I'd >> like to load each one, add a property, and save it back to the datastore. >> 400,000 times. >> >> This will obviously take a long time, so I'd really like to split it up >> into ~100 tasks that each take 1/100th of the entities (~4,000 entities) >> and perform this operation. >> >> But I really don't know how to do this using queries, and the Java >> MapReduce library is overwhelmingly complicated. >> >> So how can I create 100 tasks that each take a unique chunk of the >> entities to operate on? Is this called "sharding"? Is there a way for a >> task to say "give me entity #200,000 thru #204,000"? (My entity's keys are >> strings, which were generated by my application and generally look like >> "928348-com.example-iOS".) >> >> I'm using Java and Objectify btw. Thanks for any help or guidance!! >> >> Keith >> > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to google-appengine+unsubscr...@googlegroups.com. > To post to this group, send email to google-appengine@googlegroups.com. > Visit this group at http://groups.google.com/group/google-appengine. > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
