I should have mentioned that in addition to the 30s DEEs, we are also seeing a number of the 10s "Request was aborted" messages (the ones that occur before any user code executes).
j On Sep 2, 9:32 am, Jason C <jason.a.coll...@gmail.com> wrote: > It may have settled down for you Tim, but we are being hammered by > DeadlineExceededError right now. Log time: 8.27a Thursday 2010-09-02 > > AppID: steprep > > In the past, this problem seems to be localized to small (presumably) > clusters of apps. > > Can someone please look into our specific cluster? > > Thanks, > j > > On Sep 1, 6:06 pm, Tim Hoffman <zutes...@gmail.com> wrote: > > > > > Hi > > > It seems that the massive increase startup times has settled down. > > > But we effectively had to app instances down for about 16 hours due to > > the issue. > > It is clear the startup peformance issue is not my code as it was > > affecting a other people, but > > was not widespread enough to get on the system status radar. > > > And issue for it was > > raisedhttp://code.google.com/p/googleappengine/issues/detail?id=3667 > > > The outage was not good but to be honest the biggest problem my > > management hand with it > > was the complete absence of any response from google during this > > period. > > > Last time we had a major issue it took a week, and a large report > > prepared by myself detailing what was > > going on before I could get any response from google. The response was > > also > > limited to "This is a known issue, we are working to address it" which > > is fine in it self at least some communication > > was eventually made but it should not have taken a week and about 3 > > days of effort on my part to actually get any form of response. > > > See the final part of that saga > > here.http://groups.google.com.au/group/google-appengine/browse_thread/thre... > > > I am huge advocate of appengine and we are using it more and more, but > > the total feeling of helplessness you > > get when you have instances non functioning because they can't even > > startup for an extended period and > > no feedback what so ever makes it very hard to even manage > > expectations, and impossible to report to management > > about any form of progress in resolving the issue. > > > In addition when this happened last time, a journalist from the > > Register contacted me about the issue and asked for more information > > so they could do a story about the lack of communication from google > > on these sorts of issues. I declined, but maybe it does need some > > more visibility and least with in the app engine user community. > > > Regards > > > Tim Hoffman > > > On Sep 1, 11:54 pm, Tim Hoffman <zutes...@gmail.com> wrote: > > > > The rate of errors has dropped off somewhat. > > > > It is still happening though. > > > > T > > > > On Sep 1, 7:24 pm, Tim Hoffman <zutes...@gmail.com> wrote: > > > > > Further investigation shows these excessive startup times are being > > > > experienced by more than one instance. > > > > > Appid's qtrack-dev, q-tracker. > > > > > T > > > > > On Sep 1, 5:15 pm, Tim Hoffman <zutes...@gmail.com> wrote: > > > > > > Hi > > > > > > I am still having trouble getting instances to get past initial > > > > > import (appid q-tracker) > > > > > > I have even tried starting old versions to get around failed imports. > > > > > > for the last 10 minutes all startup attempts end in DeadlineExceeded > > > > > error. > > > > > > T > > > > > > On Sep 1, 2:45 pm, Tim Hoffman <zutes...@gmail.com> wrote: > > > > > > > Hi > > > > > > > As I have pointed out in the past its quite possible to get pockets > > > > > > of appengine instances experiencing extremely poor performance and > > > > > > this doesn't show up in the system status. (http:// > > > > > > groups.google.com.au/group/google-appengine/browse_thread/thread/ > > > > > > 23e988d494144242/8ff964f7ffcc0b37?q=#8ff964f7ffcc0b37) > > > > > > > Apps seemd to be sharded into different parts of appengine > > > > > > infrastructure based on appid. > > > > > > > I hope this problem is just transitory for tosay and I don't have to > > > > > > spend days collecting evidence again. > > > > > > > T > > > > > > > On Sep 1, 2:30 pm, Tim Hoffman <zutes...@gmail.com> wrote: > > > > > > > > Hi > > > > > > > > Except it isn't. I am still getting hit with extremeley long > > > > > > > startup > > > > > > > times. > > > > > > > I am not even getting around to accessing the datastore. Failure > > > > > > > is > > > > > > > always during instance startup. > > > > > > > > T > > > > > > > > On Sep 1, 2:27 pm, Andi Albrecht <albrecht.a...@googlemail.com> > > > > > > > wrote: > > > > > > > > > Sorry, I've missed this announcement on Downtime Notify: > > > > > > > > >http://groups.google.com/group/google-appengine-downtime-notify/brows... > > > > > > > > > As noted in my previous post, everything's fine now :) > > > > > > > > > Andi > > > > > > > > > Tim Hoffman <zutes...@gmail.com> writes: > > > > > > > > > Hi > > > > > > > > > > I am still seeing them on and off, everything might be fine > > > > > > > > > for 20min, > > > > > > > > > then bam. > > > > > > > > > I have even been having some dashboard failures. > > > > > > > > > > T > > > > > > > > > > On Sep 1, 1:55 pm, Andi Albrecht > > > > > > > > > <albrecht.a...@googlemail.com> wrote: > > > > > > > > >> Tim Hoffman <zutes...@gmail.com> writes: > > > > > > > > >> > Hi > > > > > > > > > >> > As of today I am getting a lot of DeadlineExceeded Errors > > > > > > > > >> > on a couple > > > > > > > > >> > of instances druing startup. > > > > > > > > > >> > I am not even getting past the initial imports, > > > > > > > > > >> > Nothing is being reported in system status, so it's > > > > > > > > >> > probably > > > > > > > > >> > restricted part of the appengine structure. > > > > > > > > > >> > Tim > > > > > > > > > >> Hi Tim, > > > > > > > > > >> do you still see these issues? I've seen a couple of > > > > > > > > >> different errors a > > > > > > > > >> few hours ago, but now everything looks fine again. > > > > > > > > > >> Andi -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
