Re: [google-appengine] Re: Increasing task queue quotas
Is your processing not possible with the deferred module added in 1.2.5 http://code.google.com/appengine/articles/deferred.html If you launch many tasks to process the uploads you waste a lot of CPU retrieving the data needed. With the deferred method you run for about 30 sec, then store the state and schedule to get called again. You can setup multiple deferred tasks to process 1 upload by using parameters of the function that deferred calls. 2009/11/18 Julian Namaro namarojul...@gmail.com: I'm not sure about what you want to do but just a thought: have you considered Amazon Elastic MapReduce ? It's sure doable with task queues but you're likely to encounter various limitation problems as you cite. On Nov 18, 1:46 am, James Cooper jamespcoo...@gmail.com wrote: Hi, I'm evaluating GAE suitability for one of my clients. It is a B2B app with very low organic web traffic. However, users upload lists of data that need to be processed in the background. In the absence of native MapReduce support, my plan is to use task queues. But to meet the performance needs of this application, I need to burst to relatively high levels of concurrency. My question is about the discrepancy in the GAE quotas for inbound web traffic vs. task queues. Inbound web traffic can burst to: 500 qps Task queues burst to: 20 qps In my mind, the total concurrency that GAE would provide an app is equal to: inbound qps + task queue qps + cron qps. Given that task queues are implemented as web request handlers, I see little infrastructural reason to distinguish between task queue and inbound traffic. Is it possible to request a task queue quota increase to 500 qps? If not is there a technical reason for this? thanks -- James -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=.
[google-appengine] Re: Increasing task queue quotas
Hi there, Yes, I've considered Amazon's offering, but would prefer to keep the entire system on GAE if possible to reduce complexity. thanks, -- James On Nov 17, 7:13 pm, Julian Namaro namarojul...@gmail.com wrote: I'm not sure about what you want to do but just a thought: have you considered Amazon Elastic MapReduce ? It's sure doable with task queues but you're likely to encounter various limitation problems as you cite. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=.
[google-appengine] Re: Increasing task queue quotas
Hi, My primary concern is throughput, not cost. I would be happy to pay more to gain higher concurrency and higher throughput. If a customer uploads a list with 100,000 items to process, I would like to burst to some upper limit of concurrency (higher than 20 qps at least) and process the list as quickly as possible. This seems to be something GAE could provide if the concurrency limits were raised. My question is whether Google would allow a customer to request a quota increase for task queue concurrency in situations like this, or whether that's forbidden across the board for some reason. Anyone from Google who can help answer this? thanks, -- James On Nov 18, 1:24 am, djidjadji djidja...@gmail.com wrote: Is your processing not possible with the deferred module added in 1.2.5 http://code.google.com/appengine/articles/deferred.html If you launch many tasks to process the uploads you waste a lot of CPU retrieving the data needed. With the deferred method you run for about 30 sec, then store the state and schedule to get called again. You can setup multiple deferred tasks to process 1 upload by using parameters of the function that deferred calls. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=.
[google-appengine] Re: Increasing task queue quotas
I'm not sure about what you want to do but just a thought: have you considered Amazon Elastic MapReduce ? It's sure doable with task queues but you're likely to encounter various limitation problems as you cite. On Nov 18, 1:46 am, James Cooper jamespcoo...@gmail.com wrote: Hi, I'm evaluating GAE suitability for one of my clients. It is a B2B app with very low organic web traffic. However, users upload lists of data that need to be processed in the background. In the absence of native MapReduce support, my plan is to use task queues. But to meet the performance needs of this application, I need to burst to relatively high levels of concurrency. My question is about the discrepancy in the GAE quotas for inbound web traffic vs. task queues. Inbound web traffic can burst to: 500 qps Task queues burst to: 20 qps In my mind, the total concurrency that GAE would provide an app is equal to: inbound qps + task queue qps + cron qps. Given that task queues are implemented as web request handlers, I see little infrastructural reason to distinguish between task queue and inbound traffic. Is it possible to request a task queue quota increase to 500 qps? If not is there a technical reason for this? thanks -- James -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=.
