I'm now hitting TONS of "over quota" errors, and I wasn't before.
This is really hurting my revenue today. http://www.careercup.com On Nov 13, 11:09 am, "Brett (Google)" <[EMAIL PROTECTED]> wrote: > Hi Gee, > > Definitely understood that there's been quite a bit of trouble the > last two days, especially for you and a few other apps. We're sorry > you guys have been hurting from this. I can imagine it's especially > frustrating for a growing service. > > In the specific case of this memcache maintenance, the fail-fast issue > lead to high CPU and response time problems for applications that rely > heavily on memcache. We're working to resolve this fail-fast issue so > it will not happen again. We also came up with an interim solution > that will prevent this problem from happening during future memcache > maintenances. > > Otherwise, I understand the pain you're feeling here, and relaxing > response times and CPU times would be one way to alleviate that. Our > team is working hard to improve this in general. Feel free to email > Pete, Paul, or me if you have any more questions. > > The good news is that memcache is now back up and serving. Please let > us know if you encounter any more issues! > > -Brett > > On Nov 13, 10:50 am, gee <[EMAIL PROTECTED]> wrote: > > > Hi Brett, > > > During this time, it would have been very extremely helpful if CPU/ > > response time quotas were relaxed. > > > All our cache misses are adding up, and we're now over quota. > > > We've had significant downtime over the last two days due to the > > scheduled downtimes going awry (our app was one of those taken down > > unexpectedly Tuesday night), and it's definitely hurting our users' > > experience with our growing service. Any help would be much > > appreciated. > > > Thanks > > Gee > > > On Nov 13, 10:30 am, "Brett (Google)" <[EMAIL PROTECTED]> > > wrote: > > > > Memcache calls should now be failing fast. That should significantly > > > reduce the number of request deadlines people are seeing. Otherwise, > > > we're working to bring the memcache API back online for all > > > applications as soon as possible. Thanks again for your patience, > > > everyone! > > > > On Nov 13, 10:12 am, "Brett (Google)" <[EMAIL PROTECTED]> > > > wrote: > > > > > Hi Gayle, > > > > > Yes, the *request* times out during the API call. It's not the API > > > > call itself that's timing out. But there is no effective difference to > > > > your application if you're caught in this situation (as you pointed > > > > out). I'd assume the first few calls you make to memcache during a > > > > request actually return False and None. But later on in your request > > > > handler, so much time has already passed that your API call gets > > > > interrupted by the DeadlineExceededException for the entire request. > > > > > I think for now your best bet is to catch the runtime > > > > DeadlineExceededException and immediately return with some kind of > > > > error code. We're looking into the cause of the delayed memcache calls > > > > on failure. They should be immediately failing instead of waiting up > > > > to a second to return. That's the root cause of the issue here. > > > > > I'm sorry for the trouble this is causing for your application. We're > > > > working right now to address this issue. Someone from the team will > > > > follow up when everything has been resolved. > > > > > -Brett > > > > > On Nov 13, 9:57 am, Gayle Laakmann <[EMAIL PROTECTED]> wrote: > > > > > > It was timing out within memcache.get and memcache.set - > > > > > consistently. If I understand you correctly - you're saying that > > > > > memcache isn't technically timing out, on the grounds that it would > > > > > complete if given enough time. But, the cache misses take SO much > > > > > time that it consistently timeouts during the operations of > > > > > memcache.get and memcache.set. > > > > > > Ok, sure, that may be the case. I'm afraid I don't really see the > > > > > difference between that and timing out. Either way, memcache.get and > > > > > memcache.set aren't exactly just "return false and letting your > > > > > application proceed." The fact is they are causing timeouts. > > > > > > obj = memcache.get(cache_key) > > > > > File "/base/data/home/apps/careercup/13.329261908074364999/memcache/ > > > > > __init__.py", line 362, in get > > > > > self._make_sync_call('memcache', 'Get', request, response) > > > > > > File "/base/python_lib/versions/1/google/appengine/api/ > > > > > apiproxy_stub_map.py", line 46, in MakeSyncCall > > > > > stub.MakeSyncCall(service, call, request, response) > > > > > File "/base/python_lib/versions/1/google/appengine/runtime/ > > > > > apiproxy.py", line 245, in MakeSyncCall > > > > > > rpc.Wait() > > > > > File "/base/python_lib/versions/1/google/appengine/runtime/ > > > > > apiproxy.py", line 161, in Wait > > > > > rpc_completed = _apphosting_runtime___python__apiproxy.Wait(self) > > > > > > On Nov 13, 9:45 am, Brett <[EMAIL PROTECTED]> wrote: > > > > > > > Hi Gayle, > > > > > > > I'm sorry to hear you're seeing issues. It's important to note that > > > > > > there are two types of DeadlineExceededErrors. One is defined in > > > > > > apiproxy_errors.py. The other is in runtime/__init__.py: > > > > > > >http://code.google.com/p/googleappengine/source/browse/trunk/google/a... > > > > > > > The names are the same, which makes this confusing. The latter error > > > > > > is raised when the request has been processing for too long (~10 > > > > > > seconds). Perhaps what's happening here is that the cache misses are > > > > > > causing a lot of extra work to be done, causing your application to > > > > > > hit this other deadline. > > > > > > > The traceback sent by Pratham confirms this behavior. More detail on > > > > > > how to deal with this is here, in the section entitled "The Request > > > > > > Timer": > > > > > > >http://code.google.com/appengine/docs/python/requestsandcgi.html > > > > > > > -Brett > > > > > > > On Nov 13, 9:29 am, Gayle Laakmann <[EMAIL PROTECTED]> wrote: > > > > > > > > Despite what the post at the link below says, memcache is actually > > > > > > > timing out and throwing a DeadlineExceededError. Furthermore, it > > > > > > > hardly seems accurate to say that our apps should continue serving > > > > > > > normally. The quota limits are so ridiculously low that we CAN'T > > > > > > > serve pages without caching. > > > > > > > >http://groups.google.com/group/google-appengine-downtime-notify/brows... > > > > > > > We will be taking memcache offline tomorrow morning from 9-10am > > > > > > > PST > > > > > > > (GMT-8) for routine maintenance. Calls to the memcache API will > > > > > > > *not* > > > > > > > throw exceptions but will instead return false for set() calls and > > > > > > > None for get() calls (just like any other cache miss.) > > > > > > > > Your app should continue serving normally during this period, and > > > > > > > we'll keep you updated on our progress. " > > > > > > > > Additonally, I'd like to make two suggestions: > > > > > > > 1. If you're going to take down memcache, wouldn't it make sense > > > > > > > to > > > > > > > remove the quota limits? It hardly seems fair to penalized for > > > > > > > exceeding quota when caching is disabled. > > > > > > > > 2. If you have to take down our apps for an hour, can you pick > > > > > > > something like 2am - 3am? I know you don't want to go to work at > > > > > > > 2am, > > > > > > > but it's not really ok to take down our apps for an hour during > > > > > > > prime- > > > > > > > time. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---
