Hi Eli, Thanks for the great feedback. If Task Queue Java is as evolved as Phyton that maybe the way to go.
-- superco...@gmail.com http://supercobrablogger.blogspot.com/ On Fri, Jan 14, 2011 at 12:52 PM, Eli Jones <eli.jo...@gmail.com> wrote: > You should probably start experimenting with the Taskqueue to get a better > idea how it functions and what is possible. > When you create a Task.. you can define a countdown or eta value.. so, the > task gets added.. but it is scheduled to execute at a future time (not > immediately). > So, you can just create a named, chained task that does a batch of > URLfetches asynchronously with a callback (not sure how many can be done at > once, one way to find out is to test it.).. and once it is done gathering > and putting data.. it schedules the next 5 minute batch. > http://code.google.com/appengine/docs/python/urlfetch/asynchronousrequests.html > See Nick Johnson for an overview of how to do chaining with deferred (I > personally prefer regular Taskqueue, but deferred has its positive aspects): > http://blog.notdot.net/2010/03/Task-Queue-task-chaining-done-right > You could also have a cron job that creates all of your scheduled tasks for > the day at once. Depending on if there would be no issues with some tasks > not finishing before the next 5 minute task. > > On Fri, Jan 14, 2011 at 1:24 PM, supercobra <superco...@gmail.com> wrote: >> >> Hello again, >> >> >> > Again as Eli mentioned, to offer any solutions we'd need to know what >> > you're trying to do in all those tasks. >> >> We are exploring the idea of migrating our website monitoring service >> Montastic (www.montastic.com) to GAE. Right now it works on MySQL but >> scalability becomes challenging and we need to explore a >> re-architecture of our system. GAE would be ideal if it was possible. >> So we do need to do a LOT of UrlFetch-es every 1, 2, 5 minutes. >> >> One of the challenge is to wait for 5 minutes. E.g. Fetch a URL, store >> results, wait 5 min, do it again. Since a queue will execute the task >> almost immediately (if it is empty) this would not work unless the >> queue is filled w/ a known number of tasks. >> >> >> Any suggestion welcome. >> >> >> Daniel >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to google-appengine@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
