> 1) There are no data-intensive applications that do not at some point > require batch processing. No matter how clever you are in designing > your application, at some point you will need to run a big query to > calculate statistics, change the model, clean the data, etc. If you > haven't had to do this yet you haven't run your app long enough
yes, i do agree data-intensive applications very likely needs some "data cleaning" tasks, and from times to times the model needs to be updated. to do those 'data cleaning' and model updating tasks, similar to your approaches, i use the techniques to split the large task into multiple small tasks > 2) Appengine *does* support batch processing. It just does so in a > very clunky and akward way: the task queue. You write a task that > crunches some data, watches for when it gets near the 30s deadline, > and then respawns itself to pick up where it left off. ya, and task queue for me, is a queue, which enable periodic / scheduled tasks to be executed, and can make my system to be more asynchronized (response to the users quickly and put some tasks into the queue which can be executed later). i'm not sure how the task queue makes batch processing becoming easy on GAE. (your approach is also some kind of breaking a large task into small tasks, and those smaller tasks can be executed directly without putting into a queue) so for me, GAE is still not a platform suitable for 'batch processing' > The question is not whether GAE supports batch processing - it does - > it's just a question of how clumsy the API is. It would be a lot less > clumsy if the 30s limitation were removed. as long as GAE post such limitations (30s for task in queue or 10s for normal task) , i dont think GAE is intent for real batch processing (all those techniques used by you and me are just some walk-around for housekeeping tasks, but not the real service that can be provided to general users) - eric
-- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
