[google-appengine] Re: Why are several production issues related to DeadlineExceededErrors being ignored?

2012-01-13 Thread Olivier
+1 On Jan 13, 11:21 am, Karl Rosaen wrote: > We've been hit by random DeadlineExceededErrors in the past 36 hours, so I > filed a production > ticket:http://code.google.com/p/googleappengine/issues/detail?id=6729 > > it's disconcerting to see that several similar issues have been left > ignored

[google-appengine] Re: Why are several production issues related to DeadlineExceededErrors being ignored?

2012-01-13 Thread GAEfan
+1 -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit

[google-appengine] Re: Why are several production issues related to DeadlineExceededErrors being ignored?

2012-01-14 Thread Alexis
+1 On 14 jan, 03:54, GAEfan wrote: > +1 -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@go

[google-appengine] Re: Why are several production issues related to DeadlineExceededErrors being ignored?

2012-01-15 Thread Cezary Wagner
I think that is big problem since GAE that no handle problem with customer experience of start-ups/low traffic application it kills traffic expansion. First instance loads to slow to server first customer - result is drop with 500 status customer goes away that should not occur since it block traf

[google-appengine] Re: Why are several production issues related to DeadlineExceededErrors being ignored?

2012-01-16 Thread Alexis
Thanks for the shot! Here are some comments: - isn't the limit 60sec instead of 15sec? - We indeed have the pending latency slider set to default, but out of our 12 instances we have 3 resident ones, so the slider will have little effect on our application's performance. - Requests that DEE typ

[google-appengine] Re: Why are several production issues related to DeadlineExceededErrors being ignored?

2012-01-16 Thread Alexis
Kenneth: I'm talking HRD -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com.

[google-appengine] Re: Why are several production issues related to DeadlineExceededErrors being ignored?

2012-01-16 Thread Cezary Wagner
gle-appengine@googlegroups.com > Subject: Re: [google-appengine] Re: Why are several production issues > related to DeadlineExceededErrors being ignored? > > Brandon, thanks so much for taking the time to put together the video, very > helpful. > > The key insight seem to be: time

[google-appengine] Re: Why are several production issues related to DeadlineExceededErrors being ignored?

2012-01-17 Thread Kenneth
Wow, sorry to hear that. Is it java or python? I've been on hrd now for about a month. I have the sliders set to el cheapo mode and haven't had any timeouts at all. My understanding is the startup timeout is 10 seconds. The total request timeout is 60 seconds. -- You received this message

[google-appengine] Re: Why are several production issues related to DeadlineExceededErrors being ignored?

2012-01-17 Thread Cezary Wagner
hon has its own > "Specialness" > > and what is true in traditional python is not always quite the same in > > GAE land. > > > From: google-appengine@googlegroups.com > > [mailto:google-appengine@googlegroups.com] On Behalf Of Karl Rosaen > > Sent: Monday, Jan

[google-appengine] Re: Why are several production issues related to DeadlineExceededErrors being ignored?

2012-01-17 Thread Cezary Wagner
Kenneth, The timeouts could not be 10s or 15s - since DeadlineExceededErrors occurs in my logs is about 60s - I think that 10s or 15s is myths or wrong implementation since what is sense to give 60s if it should start in 15s??? Common timeouts are: 64608ms, 70806ms, 63093ms, 64499ms, ... On Jan

[google-appengine] Re: Why are several production issues related to DeadlineExceededErrors being ignored?

2012-01-17 Thread Cezary Wagner
Behalf Of Cezary Wagner > Sent: Tuesday, January 17, 2012 1:43 AM > To: Google App Engine > Subject: [google-appengine] Re: Why are several production issues related to > DeadlineExceededErrors being ignored? > > Kenneth, > > The timeouts could not be 10s or 15s -

[google-appengine] Re: Why are several production issues related to DeadlineExceededErrors being ignored?

2012-01-18 Thread Alexis
I was in the same position in August, with our apps on M/S. The DEE errors began to appear, then were more and more frequent and finally had a high impact on our app availability. We migrated to HRD writing our own mapreduce and then all went fine (except for the costs but hey, was the price for a

[google-appengine] Re: Why are several production issues related to DeadlineExceededErrors being ignored?

2012-01-18 Thread Alexis
Cezary, I was in the same position in August, with our apps on M/S. The DEE errors began to appear, then were more and more frequent and finally had a high impact on our app availability. Our warmup requests, that only load code, were randomly taking from a few secs to more than 30sec and so trig

[google-appengine] Re: Why are several production issues related to DeadlineExceededErrors being ignored?

2012-01-18 Thread Alexis
Cezary, I was in the same position in August, with our Python apps on M/S. The DEE errors began to appear, then were more and more frequent and finally had a high impact on our app availability. Our warmup requests, that only load code, were randomly taking from a few secs to more than 30sec and so

[google-appengine] Re: Why are several production issues related to DeadlineExceededErrors being ignored?

2012-01-19 Thread Cezary Wagner
Alexis, It looks that GEA has hidden balancing/lack of resources problems or deadlocks both results in RANDOM problems. I think about moving to HRD but it looks that you suffer same on HRD the last days so it will not help till GEA will not generate RANDOM problems - I do some optimization it hel

RE: [google-appengine] Re: Why are several production issues related to DeadlineExceededErrors being ignored?

2012-01-16 Thread Brandon Wirtz
e App Engine Subject: [google-appengine] Re: Why are several production issues related to DeadlineExceededErrors being ignored? Thanks for the shot! Here are some comments: - isn't the limit 60sec instead of 15sec? - We indeed have the pending latency slider set to default, but out of o

Re: [google-appengine] Re: Why are several production issues related to DeadlineExceededErrors being ignored?

2012-01-16 Thread Kenneth
Are we talking master slave or high replication datastores? If we're talking master slave, and I'm pretty certain we are, then forget it. Google isn't going to help you. You need to migrate to hrd to avoid these problems. My life was hell with deadline errors. Then I bit the bullet and mig

Re: [google-appengine] Re: Why are several production issues related to DeadlineExceededErrors being ignored?

2012-01-16 Thread Karl Rosaen
Brandon, thanks so much for taking the time to put together the video, very helpful. The key insight seem to be: time spent in the queue waiting for a frontend counts towards the limit for a DeadlineExceedError. This seems silly - seems to me user visible latency, and framework level timeout e

Re: [google-appengine] Re: Why are several production issues related to DeadlineExceededErrors being ignored?

2012-01-16 Thread Karl Rosaen
Kenneth, Agreed that running on the master / slave datastore is a liability these days (which IMO is pretty irresponsible on GAEs part, I see no reason why major latency spikes can't be avoided even if there are advantages to the high replication datastore and we can eventually migrate. If we

RE: [google-appengine] Re: Why are several production issues related to DeadlineExceededErrors being ignored?

2012-01-16 Thread Brandon Wirtz
n GAE land. From: google-appengine@googlegroups.com [mailto:google-appengine@googlegroups.com] On Behalf Of Karl Rosaen Sent: Monday, January 16, 2012 7:32 AM To: google-appengine@googlegroups.com Subject: Re: [google-appengine] Re: Why are several production issues related to DeadlineExceededErrors bei

RE: [google-appengine] Re: Why are several production issues related to DeadlineExceededErrors being ignored?

2012-01-16 Thread Brandon Wirtz
: google-appengine@googlegroups.com [mailto:google-appengine@googlegroups.com] On Behalf Of Cezary Wagner Sent: Monday, January 16, 2012 2:37 PM To: Google App Engine Subject: [google-appengine] Re: Why are several production issues related to DeadlineExceededErrors being ignored? Brandom, I think that

Re: [google-appengine] Re: Why are several production issues related to DeadlineExceededErrors being ignored?

2012-01-17 Thread Kenneth
I always found that startup errors and datastore errors happened at the same time. I assume instances are getting the code from the datastore. For reasons unknown, the master slave datastore is broken and Google's solution is hrd. I was in the same mental position as you at the beginning of De

RE: [google-appengine] Re: Why are several production issues related to DeadlineExceededErrors being ignored?

2012-01-17 Thread Brandon Wirtz
groups.com [mailto:google-appengine@googlegroups.com] On Behalf Of Cezary Wagner Sent: Tuesday, January 17, 2012 1:37 AM To: Google App Engine Subject: [google-appengine] Re: Why are several production issues related to DeadlineExceededErrors being ignored? Brandon, I am doing service management/develo

RE: [google-appengine] Re: Why are several production issues related to DeadlineExceededErrors being ignored?

2012-01-17 Thread Brandon Wirtz
r Sent: Tuesday, January 17, 2012 1:43 AM To: Google App Engine Subject: [google-appengine] Re: Why are several production issues related to DeadlineExceededErrors being ignored? Kenneth, The timeouts could not be 10s or 15s - since DeadlineExceededErrors occurs in my logs is about 60s - I think tha