Google IO 2010 had a good description of how to handle this very problem. Have a look at the video and PDF for 'Building high- throughput data pipelines with Google App Engine' http://www.google.com/events/io/2010/sessions/high-throughput-data-pipelines-appengine.html
Dale On Jan 23, 6:00 am, Robert Kluin <robert.kl...@gmail.com> wrote: > Even with ten-minute background tasks, your datastore RPC calls are > still limited to 30 seconds. Also, if you're fetching a lot of > records and sending emails based on them, you might want to break it > up into smaller tasks anyway. Smaller chunks should also help you > handle failures in a more graceful way; if you're trying to do this as > one big job, what do you do if it fails XX% of the way through? You > may not have a good way to determine what XX is, so you can resend to > those XX% people or not send to the (1-XX)% of the people. > > Robert > > On Fri, Jan 21, 2011 at 15:00, Jay <jbaker.w...@gmail.com> wrote: > > Is it the fetch that is taking too long or the sending of the emails? > > > I cases where the jobs really do get too big (but see Ross' note), the > > typical approach is to batch up the job and use task queues. For > > example, you might have your task catch DeadlineExceeded (or whatever > > Exception that is) and smartly put the remaining 'emails' list on > > another new task. > > > On Jan 21, 12:10 pm, Ross M Karchner <rosskarch...@gmail.com> wrote: > >> Cron jobs and tasks can now take up to 10 > >> minuteshttp://code.google.com/p/googleappengine/wiki/SdkReleaseNotes > > >> On Fri, Jan 21, 2011 at 12:57 PM, Mayumi Liyanage < > > >> mayumi.liyan...@coldwin.com> wrote: > >> > Hi, I need to iterate over all the entities in the Datastore and send > >> > out emails once a day asynchronously to the actual app. > >> > Usual way to do this would be to invoke a Servlet using cron which > >> > would iterate over all the entities to send emails out. However, our > >> > data is growing at the rapid rate and sooner or later we will have a > >> > issue with 30 sec limit problem. > >> > What would be the best way to do this operation using app engine > >> > without worrying about 30 sec limit? > > >> > Can we do above using the Mapper API? If so how can we invoke a mapper > >> > from the servlet? > > >> > Thanks. > > >> > -- > >> > You received this message because you are subscribed to the Google Groups > >> > "Google App Engine" group. > >> > To post to this group, send email to google-appengine@googlegroups.com. > >> > To unsubscribe from this group, send email to > >> > google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> > >> > . > >> > For more options, visit this group at > >> >http://groups.google.com/group/google-appengine?hl=en. > > >> -- > >> Ross M Karchnerhttp://eventgrinder.com > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to google-appengine@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine+unsubscr...@googlegroups.com. > > For more options, visit this group > > athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
