I haven't done anywhere near 14,000 image transforms in the entire lifetime of the application (only 491 today for instance) so I don't think this is the reason I'm getting the OverQuotaException for image transforms.
I haven't noticed my Quota page showing "limited" for any quantity, but haven't monitored this too closely. I don't quite understand the way you've expressed the limits for memcache - do you mean there's a write limit of 56 megabytes per minute (and a read limit of 284 megabytes per minute)? If so, I'm pretty sure I'm nowhere near these limits either. At the time of the errors occurring I am reading and writing to/from memcache at a rate of no more than 2MB/minute. However, my application does nothing at all for hours or days, then I perform tests every once in a while. So these rates, whilst not high in absolute terms, are unusual for my application. Could this be relevant? Thanks, Jon On 5 Mar, 21:57, "Ikai L (Google)" <ika...@google.com> wrote: > Is there any chance you're hitting per minute quotas? > > http://code.google.com/appengine/docs/quotas.html > > Image transforms have this quota, for instance: > > 14,000 transforms/minute > > Memcache has a limit of 56/mb of writes and 284/mb of reads. > > > > On Fri, Mar 5, 2010 at 1:49 PM, Jon Blower <jon.blo...@gmail.com> wrote: > > Hi Ikai, > > > I'm afraid I don't have any unit test code easily available, but I > > could make a code sample when I have a little more time. In the > > meantime I have some more information. It's not just the memcache put > > operation that fails in this way. I've also seen similar failures of > > other API operations like datastore get: > > > com.google.apphosting.api.ApiProxy$OverQuotaException: The API > > call datastore_v3.Get() required more quota than is available > > > and calling image transform operations (to convert PNG to JPEG): > > > com.google.apphosting.api.ApiProxy$OverQuotaException: The API > > call images.Transform() required more quota than is available > > > In all cases I'm well within my daily quota for my app. During stress > > testing (using JMeter and ramping up the client threads) I see a > > consistent pattern of thirty or so successes, followed by thirty or so > > failures (of memcache put, datastore get or image transform). Then I > > see successes again, then failures and so on. > > > My interpretation of these data is as follows: > > 1) Stress test starts, all is well, all requests succeed for a while > > 2) GAE monitors the number of API calls the application makes in a 10- > > second sliding window. (I'm guessing here.) > > 3) After a while GAE detects that the app is calling the various APIs > > at a rate above a certain limit. So requests start failing. > > 4) The failed requests take longer to process than successful ones. > > So when failures start happening, the rate of calling the APIs goes > > down. > > 5) At some point the 10-second average (or whatever) dips below the > > limit, so we get successes again. > > 6) And so on... with successful requests the API calling rate goes up > > above the limit and requests start failing. > > > These are complete guesses, but would seem to fit the pattern I > > observe. Is there likely to be any truth in this? > > > Thanks, > > Jon > > > On 5 Mar, 19:40, "Ikai L (Google)" <ika...@google.com> wrote: > >> Jon, do you happen to have unit test code that reproduces this that I > >> can plug in to an app? I'd like to reproduce this. > > >> On Thu, Mar 4, 2010 at 1:31 PM, Jon Blower <jon.blo...@gmail.com> wrote: > >> > Hi, > > >> > I have a GAE-J application that involves requesting dynamically- > >> > generated images. The application has fairly high latency (~1s per > >> > request). I'm stress-testing the application using JMeter, hitting > >> > GAE with four simultaneous threads, each looping through a fixed set > >> > of requests. I get a consistent pattern, in which the first twenty or > >> > thirty requests succeed, then every subsequent request fails. The > >> > server log shows that the failures occur during a memcache put > >> > operation: > > >> > "com.google.appengine.api.memcache.stdimpl.GCacheException: Policy > >> > prevented put operation > >> > at > >> > com.google.appengine.api.memcache.stdimpl.GCache.put(GCache.java: > >> > 165)" > > >> > If I stop the stress test for a couple of minutes, then restart it I > >> > get the same pattern: success for the first 20-30 requests, then this > >> > failure thereafter. It's not the same requests that fail each time. > >> > Also, the data I'm storing in the memcache is always 1MB or less. My > >> > memcache set policy is the default of SET_ALWAYS. > > >> > Is there some policy restricting the number of memcache puts I can do > >> > per second perhaps? Or is there a thread safety issue? > > >> > Here are a few more details about my application. I have stored high- > >> > resolution images in the persistent store by breaking them up into > >> > chunks of size 1MB or less. I have layered memcache above the > >> > persistent store. So I have code that looks like this: > > >> > public byte[] requestChunk(String chunkId) { > >> > byte[] chunk = searchMemcache(chunkId); > >> > if (chunk != null) return chunk; > >> > chunk = searchPersistentStore(chunkId); > >> > if (chunk != null) { > >> > putChunkInMemcache(chunkId, chunk); // *** This is where the > >> > errors come from! *** > >> > return chunk; > >> > } > >> > return null; > >> > } > > >> > It's the putChunkInMemcache() method that fails during the stress > >> > test. > > >> > Any guidance much appreciated! > >> > Jon > > >> > -- > >> > You received this message because you are subscribed to the Google > >> > Groups "Google App Engine for Java" group. > >> > To post to this group, send email to > >> > google-appengine-j...@googlegroups.com. > >> > To unsubscribe from this group, send email to > >> > google-appengine-java+unsubscr...@googlegroups.com. > >> > For more options, visit this group > >> > athttp://groups.google.com/group/google-appengine-java?hl=en. > > >> -- > >> Ikai Lan > >> Developer Programs Engineer, Google App > >> Enginehttp://googleappengine.blogspot.com|http://twitter.com/app_engine > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine for Java" group. > > To post to this group, send email to google-appengine-j...@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine-java+unsubscr...@googlegroups.com. > > For more options, visit this group > > athttp://groups.google.com/group/google-appengine-java?hl=en. > > -- > Ikai Lan > Developer Programs Engineer, Google App > Enginehttp://googleappengine.blogspot.com|http://twitter.com/app_engine -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
