On closer inspection of my logs, it appears that I might be hitting my per-minute limits for read/write to the datastore, and possibly memcache. Although I'm nowhere near the limit of 4800 calls/minute to the image manipulation API, I might well be hitting the 5 megabyte per minute limit for that service. Apologies for my misunderstanding.
If this is the case, what's the recommended way to deal with this? Should I make repeated requests to the quota-ed service (perhaps with an exponential backoff) until I get success? This will increase the latency of my application, but I would prefer this to seeing requests fail. Thanks, Jon On Mar 5, 11:04 pm, Jon Blower <jon.blo...@gmail.com> wrote: > I haven't done anywhere near 14,000 image transforms in the entire > lifetime of the application (only 491 today for instance) so I don't > think this is the reason I'm getting the OverQuotaException for image > transforms. > > I haven't noticed my Quota page showing "limited" for any quantity, > but haven't monitored this too closely. > > I don't quite understand the way you've expressed the limits for > memcache - do you mean there's a write limit of 56 megabytes per > minute (and a read limit of 284 megabytes per minute)? If so, I'm > pretty sure I'm nowhere near these limits either. At the time of the > errors occurring I am reading and writing to/from memcache at a rate > of no more than 2MB/minute. > > However, my application does nothing at all for hours or days, then I > perform tests every once in a while. So these rates, whilst not high > in absolute terms, are unusual for my application. Could this be > relevant? > > Thanks, Jon > > On 5 Mar, 21:57, "Ikai L (Google)" <ika...@google.com> wrote: > > > Is there any chance you're hitting per minute quotas? > > >http://code.google.com/appengine/docs/quotas.html > > > Image transforms have this quota, for instance: > > > 14,000 transforms/minute > > > Memcache has a limit of 56/mb of writes and 284/mb of reads. > > > On Fri, Mar 5, 2010 at 1:49 PM, Jon Blower <jon.blo...@gmail.com> wrote: > > > Hi Ikai, > > > > I'm afraid I don't have any unit test code easily available, but I > > > could make a code sample when I have a little more time. In the > > > meantime I have some more information. It's not just the memcache put > > > operation that fails in this way. I've also seen similar failures of > > > other API operations like datastore get: > > > > com.google.apphosting.api.ApiProxy$OverQuotaException: The API > > > call datastore_v3.Get() required more quota than is available > > > > and calling image transform operations (to convert PNG to JPEG): > > > > com.google.apphosting.api.ApiProxy$OverQuotaException: The API > > > call images.Transform() required more quota than is available > > > > In all cases I'm well within my daily quota for my app. During stress > > > testing (using JMeter and ramping up the client threads) I see a > > > consistent pattern of thirty or so successes, followed by thirty or so > > > failures (of memcache put, datastore get or image transform). Then I > > > see successes again, then failures and so on. > > > > My interpretation of these data is as follows: > > > 1) Stress test starts, all is well, all requests succeed for a while > > > 2) GAE monitors the number of API calls the application makes in a 10- > > > second sliding window. (I'm guessing here.) > > > 3) After a while GAE detects that the app is calling the various APIs > > > at a rate above a certain limit. So requests start failing. > > > 4) The failed requests take longer to process than successful ones. > > > So when failures start happening, the rate of calling the APIs goes > > > down. > > > 5) At some point the 10-second average (or whatever) dips below the > > > limit, so we get successes again. > > > 6) And so on... with successful requests the API calling rate goes up > > > above the limit and requests start failing. > > > > These are complete guesses, but would seem to fit the pattern I > > > observe. Is there likely to be any truth in this? > > > > Thanks, > > > Jon > > > > On 5 Mar, 19:40, "Ikai L (Google)" <ika...@google.com> wrote: > > >> Jon, do you happen to have unit test code that reproduces this that I > > >> can plug in to an app? I'd like to reproduce this. > > > >> On Thu, Mar 4, 2010 at 1:31 PM, Jon Blower <jon.blo...@gmail.com> wrote: > > >> > Hi, > > > >> > I have a GAE-J application that involves requesting dynamically- > > >> > generated images. The application has fairly high latency (~1s per > > >> > request). I'm stress-testing the application using JMeter, hitting > > >> > GAE with four simultaneous threads, each looping through a fixed set > > >> > of requests. I get a consistent pattern, in which the first twenty or > > >> > thirty requests succeed, then every subsequent request fails. The > > >> > server log shows that the failures occur during a memcache put > > >> > operation: > > > >> > "com.google.appengine.api.memcache.stdimpl.GCacheException: Policy > > >> > prevented put operation > > >> > at > > >> > com.google.appengine.api.memcache.stdimpl.GCache.put(GCache.java: > > >> > 165)" > > > >> > If I stop the stress test for a couple of minutes, then restart it I > > >> > get the same pattern: success for the first 20-30 requests, then this > > >> > failure thereafter. It's not the same requests that fail each time. > > >> > Also, the data I'm storing in the memcache is always 1MB or less. My > > >> > memcache set policy is the default of SET_ALWAYS. > > > >> > Is there some policy restricting the number of memcache puts I can do > > >> > per second perhaps? Or is there a thread safety issue? > > > >> > Here are a few more details about my application. I have stored high- > > >> > resolution images in the persistent store by breaking them up into > > >> > chunks of size 1MB or less. I have layered memcache above the > > >> > persistent store. So I have code that looks like this: > > > >> > public byte[] requestChunk(String chunkId) { > > >> > byte[] chunk = searchMemcache(chunkId); > > >> > if (chunk != null) return chunk; > > >> > chunk = searchPersistentStore(chunkId); > > >> > if (chunk != null) { > > >> > putChunkInMemcache(chunkId, chunk); // *** This is where the > > >> > errors come from! *** > > >> > return chunk; > > >> > } > > >> > return null; > > >> > } > > > >> > It's the putChunkInMemcache() method that fails during the stress > > >> > test. > > > >> > Any guidance much appreciated! > > >> > Jon > > > >> > -- > > >> > You received this message because you are subscribed to the Google > > >> > Groups "Google App Engine for Java" group. > > >> > To post to this group, send email to > > >> > google-appengine-j...@googlegroups.com. > > >> > To unsubscribe from this group, send email to > > >> > google-appengine-java+unsubscr...@googlegroups.com. > > >> > For more options, visit this group > > >> > athttp://groups.google.com/group/google-appengine-java?hl=en. > > > >> -- > > >> Ikai Lan > > >> Developer Programs Engineer, Google App > > >> Enginehttp://googleappengine.blogspot.com|http://twitter.com/app_engine > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "Google App Engine for Java" group. > > > To post to this group, send email to > > > google-appengine-j...@googlegroups.com. > > > To unsubscribe from this group, send email to > > > google-appengine-java+unsubscr...@googlegroups.com. > > > For more options, visit this group > > > athttp://groups.google.com/group/google-appengine-java?hl=en. > > > -- > > Ikai Lan > > Developer Programs Engineer, Google App > > Enginehttp://googleappengine.blogspot.com|http://twitter.com/app_engine -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
