Some of these quotas can be increased (500 req/s) via the billing issues form:
http://code.google.com/appengine/kb/billing.html On Sun, Mar 7, 2010 at 12:55 PM, Jon Blower <jon.blo...@gmail.com> wrote: > On closer inspection of my logs, it appears that I might be hitting my > per-minute limits for read/write to the datastore, and possibly > memcache. Although I'm nowhere near the limit of 4800 calls/minute to > the image manipulation API, I might well be hitting the 5 megabyte per > minute limit for that service. Apologies for my misunderstanding. > > If this is the case, what's the recommended way to deal with this? > Should I make repeated requests to the quota-ed service (perhaps with > an exponential backoff) until I get success? This will increase the > latency of my application, but I would prefer this to seeing requests > fail. > > Thanks, Jon > > On Mar 5, 11:04 pm, Jon Blower <jon.blo...@gmail.com> wrote: >> I haven't done anywhere near 14,000 image transforms in the entire >> lifetime of the application (only 491 today for instance) so I don't >> think this is the reason I'm getting the OverQuotaException for image >> transforms. >> >> I haven't noticed my Quota page showing "limited" for any quantity, >> but haven't monitored this too closely. >> >> I don't quite understand the way you've expressed the limits for >> memcache - do you mean there's a write limit of 56 megabytes per >> minute (and a read limit of 284 megabytes per minute)? If so, I'm >> pretty sure I'm nowhere near these limits either. At the time of the >> errors occurring I am reading and writing to/from memcache at a rate >> of no more than 2MB/minute. >> >> However, my application does nothing at all for hours or days, then I >> perform tests every once in a while. So these rates, whilst not high >> in absolute terms, are unusual for my application. Could this be >> relevant? >> >> Thanks, Jon >> >> On 5 Mar, 21:57, "Ikai L (Google)" <ika...@google.com> wrote: >> >> > Is there any chance you're hitting per minute quotas? >> >> >http://code.google.com/appengine/docs/quotas.html >> >> > Image transforms have this quota, for instance: >> >> > 14,000 transforms/minute >> >> > Memcache has a limit of 56/mb of writes and 284/mb of reads. >> >> > On Fri, Mar 5, 2010 at 1:49 PM, Jon Blower <jon.blo...@gmail.com> wrote: >> > > Hi Ikai, >> >> > > I'm afraid I don't have any unit test code easily available, but I >> > > could make a code sample when I have a little more time. In the >> > > meantime I have some more information. It's not just the memcache put >> > > operation that fails in this way. I've also seen similar failures of >> > > other API operations like datastore get: >> >> > > com.google.apphosting.api.ApiProxy$OverQuotaException: The API >> > > call datastore_v3.Get() required more quota than is available >> >> > > and calling image transform operations (to convert PNG to JPEG): >> >> > > com.google.apphosting.api.ApiProxy$OverQuotaException: The API >> > > call images.Transform() required more quota than is available >> >> > > In all cases I'm well within my daily quota for my app. During stress >> > > testing (using JMeter and ramping up the client threads) I see a >> > > consistent pattern of thirty or so successes, followed by thirty or so >> > > failures (of memcache put, datastore get or image transform). Then I >> > > see successes again, then failures and so on. >> >> > > My interpretation of these data is as follows: >> > > 1) Stress test starts, all is well, all requests succeed for a while >> > > 2) GAE monitors the number of API calls the application makes in a 10- >> > > second sliding window. (I'm guessing here.) >> > > 3) After a while GAE detects that the app is calling the various APIs >> > > at a rate above a certain limit. So requests start failing. >> > > 4) The failed requests take longer to process than successful ones. >> > > So when failures start happening, the rate of calling the APIs goes >> > > down. >> > > 5) At some point the 10-second average (or whatever) dips below the >> > > limit, so we get successes again. >> > > 6) And so on... with successful requests the API calling rate goes up >> > > above the limit and requests start failing. >> >> > > These are complete guesses, but would seem to fit the pattern I >> > > observe. Is there likely to be any truth in this? >> >> > > Thanks, >> > > Jon >> >> > > On 5 Mar, 19:40, "Ikai L (Google)" <ika...@google.com> wrote: >> > >> Jon, do you happen to have unit test code that reproduces this that I >> > >> can plug in to an app? I'd like to reproduce this. >> >> > >> On Thu, Mar 4, 2010 at 1:31 PM, Jon Blower <jon.blo...@gmail.com> wrote: >> > >> > Hi, >> >> > >> > I have a GAE-J application that involves requesting dynamically- >> > >> > generated images. The application has fairly high latency (~1s per >> > >> > request). I'm stress-testing the application using JMeter, hitting >> > >> > GAE with four simultaneous threads, each looping through a fixed set >> > >> > of requests. I get a consistent pattern, in which the first twenty or >> > >> > thirty requests succeed, then every subsequent request fails. The >> > >> > server log shows that the failures occur during a memcache put >> > >> > operation: >> >> > >> > "com.google.appengine.api.memcache.stdimpl.GCacheException: Policy >> > >> > prevented put operation >> > >> > at >> > >> > com.google.appengine.api.memcache.stdimpl.GCache.put(GCache.java: >> > >> > 165)" >> >> > >> > If I stop the stress test for a couple of minutes, then restart it I >> > >> > get the same pattern: success for the first 20-30 requests, then this >> > >> > failure thereafter. It's not the same requests that fail each time. >> > >> > Also, the data I'm storing in the memcache is always 1MB or less. My >> > >> > memcache set policy is the default of SET_ALWAYS. >> >> > >> > Is there some policy restricting the number of memcache puts I can do >> > >> > per second perhaps? Or is there a thread safety issue? >> >> > >> > Here are a few more details about my application. I have stored high- >> > >> > resolution images in the persistent store by breaking them up into >> > >> > chunks of size 1MB or less. I have layered memcache above the >> > >> > persistent store. So I have code that looks like this: >> >> > >> > public byte[] requestChunk(String chunkId) { >> > >> > byte[] chunk = searchMemcache(chunkId); >> > >> > if (chunk != null) return chunk; >> > >> > chunk = searchPersistentStore(chunkId); >> > >> > if (chunk != null) { >> > >> > putChunkInMemcache(chunkId, chunk); // *** This is where the >> > >> > errors come from! *** >> > >> > return chunk; >> > >> > } >> > >> > return null; >> > >> > } >> >> > >> > It's the putChunkInMemcache() method that fails during the stress >> > >> > test. >> >> > >> > Any guidance much appreciated! >> > >> > Jon >> >> > >> > -- >> > >> > You received this message because you are subscribed to the Google >> > >> > Groups "Google App Engine for Java" group. >> > >> > To post to this group, send email to >> > >> > google-appengine-j...@googlegroups.com. >> > >> > To unsubscribe from this group, send email to >> > >> > google-appengine-java+unsubscr...@googlegroups.com. >> > >> > For more options, visit this group >> > >> > athttp://groups.google.com/group/google-appengine-java?hl=en. >> >> > >> -- >> > >> Ikai Lan >> > >> Developer Programs Engineer, Google App >> > >> Enginehttp://googleappengine.blogspot.com|http://twitter.com/app_engine >> >> > > -- >> > > You received this message because you are subscribed to the Google >> > > Groups "Google App Engine for Java" group. >> > > To post to this group, send email to >> > > google-appengine-j...@googlegroups.com. >> > > To unsubscribe from this group, send email to >> > > google-appengine-java+unsubscr...@googlegroups.com. >> > > For more options, visit this group >> > > athttp://groups.google.com/group/google-appengine-java?hl=en. >> >> > -- >> > Ikai Lan >> > Developer Programs Engineer, Google App >> > Enginehttp://googleappengine.blogspot.com|http://twitter.com/app_engine > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to google-appengine-j...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine-java+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > > -- Ikai Lan Developer Programs Engineer, Google App Engine http://googleappengine.blogspot.com | http://twitter.com/app_engine -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
