Is your processing not possible with the deferred module added in 1.2.5 http://code.google.com/appengine/articles/deferred.html
If you launch many tasks to process the uploads you waste a lot of CPU retrieving the data needed. With the deferred method you run for about 30 sec, then store the state and schedule to get called again. You can setup multiple deferred tasks to process 1 upload by using parameters of the function that deferred calls. 2009/11/18 Julian Namaro <namarojul...@gmail.com>: > I'm not sure about what you want to do but just a thought: have you > considered Amazon Elastic MapReduce ? > It's sure doable with task queues but you're likely to encounter > various limitation problems as you cite. > > > On Nov 18, 1:46 am, James Cooper <jamespcoo...@gmail.com> wrote: >> Hi, >> >> I'm evaluating GAE suitability for one of my clients. It is a B2B app >> with very low organic web traffic. However, users upload lists of >> data that need to be processed in the background. >> >> In the absence of native MapReduce support, my plan is to use task >> queues. But to meet the performance needs of this application, I need >> to burst to relatively high levels of concurrency. >> >> My question is about the discrepancy in the GAE quotas for inbound web >> traffic vs. task queues. >> >> Inbound web traffic can burst to: 500 qps >> Task queues burst to: 20 qps >> >> In my mind, the total concurrency that GAE would provide an app is >> equal to: >> inbound qps + task queue qps + cron qps. >> >> Given that task queues are implemented as web request handlers, I see >> little infrastructural reason to distinguish between task queue and >> inbound traffic. >> >> Is it possible to request a task queue quota increase to 500 qps? If >> not is there a technical reason for this? >> >> thanks >> >> -- James > > -- > > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appeng...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=. > > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=.
