Takashi, Thanks for a meaningful suggestion. I'm on my way to implement timeouts / deadlines for my memcache API calls. However, I wonder what the rational is behind not having SLAs for memcaches. SLA can be made lenient, but they should still exist. Otherwise how do we hold you accountable for the features you provide. Not having any SLAs sounds like you are absolving yourself of all responsibilities. Memcache, as a feature, should provide a "cheaper" access to underlying data, by definition, in exchange for loss of reliability. If Memcache API calls have huge latencies associated with them, then its not cheap anymore. And if the net cost of using memcache goes higher than the cost of using datastore, why does the memcache even exist? Additionally, I guess if memcache has strictly temporary spikes in latency, then that's understandable too, but if an issue persists through an entire day, then it calls into question the very existence of memcache. And finally, the issues described in this email chain are frighteningly correlated with the latest 1.6.6 upgrade of google app engine.
Here's the production issue I have logged, starred by several others, and nothing concrete has been done about it: http://code.google.com/p/googleappengine/issues/detail?id=7554 On Tue, May 22, 2012 at 5:19 PM, Takashi Matsuo <tmat...@google.com> wrote: > Thanks for reporting guys! > > Ideally, the memcache service should have steady latencies, but > unfortunately it is not as stable as HR datastore. Additionally, our > SLA is not covering the memcache service now. In general, we're > working hard for improving stability of our service. However, of > course we prioritize SLA-covered serviced first. > > Thus, you may want to file a feature request for making the memcache > service covered by SLA. That way, others can show their > interests/business needs by staring it, and if there's many stars on > the issue, our engineering team will eventually prioritize providing > SLA-covered memcache service. > > For the time being, you can put appropriate deadline on your memcache > call to prevent those kind of latency problems happening. Python > memcache Client class has some methods like get_multi_async and > set_multi_async that you can put a deadline(by specifying an rpc > object). > > For more details, please see: > > https://developers.google.com/appengine/docs/python/memcache/clientclass#Client_get_multi_async > > -- Takashi > > On Wed, May 23, 2012 at 6:36 AM, Rishi Arora <rishi.ar...@ship-rack.com> > wrote: > > Here's my appstats output for a specific request showing my problem is > > squarely related to memcache: > > > > > > > > > > On Tue, May 22, 2012 at 4:33 PM, Rishi Arora <rishi.ar...@ship-rack.com> > > wrote: > >> > >> Hmmm. That means the problem isn't specific to memcache. And that > means > >> it might take longer for Google to fix this, if indeed they determine > >> there's a problem. > >> > >> > >> On Tue, May 22, 2012 at 4:15 PM, Robert Morgan <robe...@coastside.net> > >> wrote: > >>> > >>> My graph is attached -- interesting, eh? > >>> I don't think we're using memcache in this instance. > >>> > >>> :R > >>> > >>> > >>> On Tue, May 22, 2012 at 2:04 PM, Rishi Arora < > rishi.ar...@ship-rack.com> > >>> wrote: > >>>> > >>>> Here's what my milliseconds/request looks like, although the > #instances > >>>> follows a similar pattern, and translates directly into higher costs > (5x > >>>> normal usage today) > >>>> > >>>> > >>>> On Tue, May 22, 2012 at 4:01 PM, Rishi Arora < > rishi.ar...@ship-rack.com> > >>>> wrote: > >>>>> > >>>>> Perhaps similar to what I'm seeing. Although I was able to use the > >>>>> appstats tool to determine that the deadlines were being missed > specifically > >>>>> because of memcache API calls. Perhaps you can determine from > appstats if > >>>>> its the same for you? > >>>>> > >>>>> If so, please star this issue that I logged: > >>>>> > >>>>> code.google.com/p/googleappengine/issues/detail?id=7554 > >>>>> > >>>>> > >>>>> > >>>>> On Tue, May 22, 2012 at 2:02 PM, Robert Morgan < > gummywubb...@gmail.com> > >>>>> wrote: > >>>>>> > >>>>>> Beginning about 18 hours ago, the milliseconds/Request on one of our > >>>>>> apps began to ramp and is now over 100 seconds for a trivial > response (which > >>>>>> historically takes 2 seconds). Our other apps are fine, and the AE > status > >>>>>> page is clear. > >>>>>> > >>>>>> It's an app that we use for development, it's been around for ages > and > >>>>>> still runs the Master-Slave datastore. Python. The ramping just > started -- > >>>>>> we had not deployed a new server version in about a week, and our > client > >>>>>> usage remains low and patterns haven't changed. > >>>>>> > >>>>>> I've poked at it trying to see if I can determine some threshold, > but > >>>>>> even a very simple return now fails due to DeadlineExceeded. > >>>>>> > >>>>>> This really feels like it's a system issue -- any ideas on what I > can > >>>>>> do next? > >>>>>> > >>>>>> Thanks, > >>>>>> :R > >>>>>> > >>>>>> -- > >>>>>> You received this message because you are subscribed to the Google > >>>>>> Groups "Google App Engine" group. > >>>>>> To view this discussion on the web visit > >>>>>> https://groups.google.com/d/msg/google-appengine/-/rYY5Gw9UuE8J. > >>>>>> To post to this group, send email to > >>>>>> google-appengine@googlegroups.com. > >>>>>> To unsubscribe from this group, send email to > >>>>>> google-appengine+unsubscr...@googlegroups.com. > >>>>>> For more options, visit this group at > >>>>>> http://groups.google.com/group/google-appengine?hl=en. > >>>>> > >>>>> > >>>> > >>>> -- > >>>> You received this message because you are subscribed to the Google > >>>> Groups "Google App Engine" group. > >>>> To post to this group, send email to > google-appengine@googlegroups.com. > >>>> To unsubscribe from this group, send email to > >>>> google-appengine+unsubscr...@googlegroups.com. > >>>> For more options, visit this group at > >>>> http://groups.google.com/group/google-appengine?hl=en. > >>> > >>> > >>> -- > >>> You received this message because you are subscribed to the Google > Groups > >>> "Google App Engine" group. > >>> To post to this group, send email to google-appengine@googlegroups.com > . > >>> To unsubscribe from this group, send email to > >>> google-appengine+unsubscr...@googlegroups.com. > >>> For more options, visit this group at > >>> http://groups.google.com/group/google-appengine?hl=en. > >> > >> > > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to google-appengine@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine+unsubscr...@googlegroups.com. > > For more options, visit this group at > > http://groups.google.com/group/google-appengine?hl=en. > > > > -- > Takashi Matsuo | Developer Advocate | tmat...@google.com > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
