I'm not sure about what you want to do but just a thought: have you considered Amazon Elastic MapReduce ? It's sure doable with task queues but you're likely to encounter various limitation problems as you cite.
On Nov 18, 1:46 am, James Cooper <jamespcoo...@gmail.com> wrote: > Hi, > > I'm evaluating GAE suitability for one of my clients. It is a B2B app > with very low organic web traffic. However, users upload lists of > data that need to be processed in the background. > > In the absence of native MapReduce support, my plan is to use task > queues. But to meet the performance needs of this application, I need > to burst to relatively high levels of concurrency. > > My question is about the discrepancy in the GAE quotas for inbound web > traffic vs. task queues. > > Inbound web traffic can burst to: 500 qps > Task queues burst to: 20 qps > > In my mind, the total concurrency that GAE would provide an app is > equal to: > inbound qps + task queue qps + cron qps. > > Given that task queues are implemented as web request handlers, I see > little infrastructural reason to distinguish between task queue and > inbound traffic. > > Is it possible to request a task queue quota increase to 500 qps? If > not is there a technical reason for this? > > thanks > > -- James -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=.
