There is a lot of stuff on the roadmap which might represent a different approach to the problem
http://code.google.com/appengine/docs/roadmap.html especially - - Background servers capable of running for longer than 30s - Support for mapping operations across datasets and maybe these - - Ability to reserve instances to reduce application loading overhead (direct long running tasks to reserved instances?) - Ability to select different availability vs. latency options for Datastore (do we have this already?) - Datastore dump and restore facility (so we don't need to use bulk loader) - Raise request/response size limits for some APIs But without a clear indication of timing, architecture direction, or what the criteria for the request limiting behaviour is, it is difficult to know what the best practice is for people building apps right now. I'd guess that we'll get a good look at some of this in about a month (google i/o) - but would it be fair to say that right now, there really isn't a good option for time consuming tasks? Seems there's two possibilities: - long running deferred, causing request limiting - short running deferred, pushing task queue limits Does that sound about right? Also, what about the bulk loader - is it fair to assume that using the bulk loader will cause request limits to be applied to an app? Appreciate any additional info - cheers, Colin On Apr 16, 3:40 pm, bFlood <bflood...@gmail.com> wrote: > this is exactly what I'm seeing with the Task Queue as of yesterday. > it was ripping fast 2 days ago, I was getting close to 150 features/ > second upload/index. now this has dropped considerably and there are > many timeouts on the TQ after only 10secs (startup time, not even > hitting my code). > > http://groups.google.com/group/google-appengine/browse_thread/thread/... > > On Apr 16, 10:20 am, hawkett <hawk...@gmail.com> wrote: > > > > > > > Hi Nick, > > > This is a little disconcerting - would it be fair to say that if I > > were using the pattern you documented > > herehttp://code.google.com/appengine/articles/deferred.html, > > then my latencies would not be low (since I am pushing the > > DeadlineExceededError on every deferred operation), and my app will > > become request limited? Perhaps this pattern should be processing only > > one or two records at a time (and then risking the task limit)? > > Perhaps it does not apply to requests initiated from the task queue? > > Would be good to get some clarification on what we can and can't do. > > Cheers, > > > Colin > > > On Apr 10, 4:18 pm, "Nick Johnson (Google)" <nick.john...@google.com> > > wrote: > > > > Hi Peter, > > > > The simultaneous dynamic request limit was eliminated in a recent SDK. It > > > has been replaced with the flexible provisioning system detailed in the > > > FAQ > > > entry you quote. > > > > It's correct that as long as you keep your latencies low, you should never > > > run into any sort of request limit. > > > > -Nick Johnson > > > > On Sat, Apr 10, 2010 at 2:12 PM, peterk <peter.ke...@gmail.com> wrote: > > > > I've been using and watching appengine since, well, the beginning, and > > > > only by a random Google about an unrelated topic did I find out about > > > > this notion of a cap on 'simultaneous active requests'. I came across > > > > discussions where people were saying that there is a cap of 30 > > > > simultaneous active requests, citing the Quota page and so on. > > > > > But when I go to the Quota page - there's not a single mention of any > > > > such limit. Why not? Has something changed? > > > > > Reading the quotas on that page it's easy to conclude - naively > > > > perhaps in hindsight - that 500 requests/second is 500 requests/second > > > > irrespective of the average execution time of those requests. > > > > > There is no mention of any assumption on that page about the time > > > > taken to serve a request or anything else that goes into the > > > > calculation for 500 requests/second. > > > > > After some further searching I finally found mention of this limit in > > > > the FAQs, but here it's still quite vague - and again, no specific > > > > mention of this 30 simultaneous request limit. It says: > > > > > "However, App Engine reserves automatic scaling capacity for > > > > applications with low latency, where the application responds to > > > > requests in less than one second. Applications with very high latency > > > > (over one second per request for many requests) are limited by the > > > > system, and require a special exemption in order to have a large > > > > number of simultaneous dynamic requests. If your application has a > > > > strong need for a high throughput of long-running requests, you can > > > > request an exemption from the simultaneous dynamic request limit. The > > > > vast majority of applications do not require any exemption." > > > > > Which doesn't imply a hard limit, but rather that as long as your > > > > (average?) request fulfilment time is below one second you shouldn't > > > > have to worry about this. To be very exacting, if I keep my average > > > > request time under 1 second I should *never* see an error relating to > > > > this cap? This is a bit different than the talk of 30 simultaneous > > > > requests max - at that level with the max of 500 requests a second, > > > > you'd be limited to an average request time of 60ms! > > > > > So what is the story here? Why is this all being talked about in the > > > > shadows of message boards with no clear policy? Why isn't this > > > > limitation in the quota page? I thought I had GAE all figured out but > > > > then this ball gets thrown into the mix out of left field. > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups > > > > "Google App Engine" group. > > > > To post to this group, send email to google-appeng...@googlegroups.com. > > > > To unsubscribe from this group, send email to > > > > google-appengine+unsubscr...@googlegroups.com<google-appengine%2Bunsubscrib > > > > e...@googlegroups.com> > > > > . > > > > For more options, visit this group at > > > >http://groups.google.com/group/google-appengine?hl=en. > > > > -- > > > Nick Johnson, Developer Programs Engineer, App Engine Google Ireland Ltd. > > > :: > > > Registered in Dublin, Ireland, Registration Number: 368047 > > > Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: > > > 368047 > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to google-appeng...@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine+unsubscr...@googlegroups.com. > > For more options, visit this group > > athttp://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appeng...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group > athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
