I think it is pretty standard practice, and it can be quite fast when implemented correctly.
Check out Brett Slatkin's IO talk: http://code.google.com/events/io/2010/sessions/high-throughput-data-pipelines-appengine.html And the 'mapper' implementation: http://code.google.com/p/appengine-mapreduce/ Robert On Mon, Jul 12, 2010 at 8:13 PM, Phil McDonnell <phil.a.mcdonn...@gmail.com> wrote: > I see how this is possible, but it feels very clunky. Is this standard > practice? > Thanks, > Phil > > On Mon, Jul 12, 2010 at 7:09 PM, Pieter Coucke <pieter.cou...@onthoo.com> > wrote: >> >> For a similar problem I use a task A which fetches 10 entities, creates >> subtasks B for them and then creates a new task A with the datastore cursor >> as a parameter so this task can continue creating subtasks. >> >> >> >> On Mon, Jul 12, 2010 at 1:31 AM, Phil McDonnell >> <phil.a.mcdonn...@gmail.com> wrote: >>> >>> I know there are a number of different frameworks and mechanisms to >>> process data offline on GAE. I need to have a daily job that scans one of >>> my datastore tables and computes an updated profile for all my users. What >>> would be the best mechanism to do this? Are task queues appropriate? >>> Should I be using some sort of mapreduce? If I should use some sort of >>> mapreduce framework, how does that integrate with GAE? My user base right >>> now is small, but I intend to scale this. >>> Thanks, >>> Phil >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Google App Engine" group. >>> To post to this group, send email to google-appeng...@googlegroups.com. >>> To unsubscribe from this group, send email to >>> google-appengine+unsubscr...@googlegroups.com. >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine?hl=en. >> >> >> >> -- >> Pieter Coucke >> Onthoo BVBA >> http://www.onthoo.com >> http://www.koopjeszoeker.be >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to google-appeng...@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appeng...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
