RE: [google-appengine] Re: APP DOWN: App suddenly no longer starts, no code changes

2012-09-13 Thread Drake
Doing Maintenance outside a scheduled window with no warning is dumb. We use Jenkins in my shop. And we do production testing across a subset of our install base before we push everyone to it. In short we test the hell before anything goes live, then we bash the hell out of it to see that it

Re: [google-appengine] Re: APP DOWN: App suddenly no longer starts, no code changes

2012-09-13 Thread alex
Actually, talking about open stacks, I've been playing with AppFog recently and I've gotta say my test apps running on Java, Ruby and Python runtimes have impressive performance and throughput. I was pretty skeptical at the beginning (I did have a look at Cloudfoundry code base a year back

Re: [google-appengine] Re: APP DOWN: App suddenly no longer starts, no code changes

2012-09-13 Thread Jeff Schnitzer
I should probably also chime in with a positive note - on the bright side, I haven't heard anyone complain about the datastore in a very long time. The HRD does seem to deliver on its promise. Now we just need all the rest of the infrastructure to come up to this same level of robustness! Jeff

Re: [google-appengine] Re: APP DOWN: App suddenly no longer starts, no code changes

2012-09-13 Thread Richard Watson
Christina wrote: *We have resolved the first issue by fixing the underlying bug.* I think one good option for longer-term stability is one we've identified before - we need a stable release channel where Google won't alter the environment until it's thoroughly tested, meaning deployed to

Re: [google-appengine] Re: APP DOWN: App suddenly no longer starts, no code changes

2012-09-13 Thread Andy Stevko
re: communication Perhaps the app engine team can learn from the pain suffering customers endured with another vendor's service outage rather than bleeding us all over again. http://blog.rightscale.com/2011/04/25/amazon-ec2-outage-summary-and-lessons-learned/ the biggest failure in this event

[google-appengine] Re: APP DOWN: App suddenly no longer starts, no code changes

2012-09-13 Thread hyperflame
Christina, Let me take a wild guess at what happened internally. A few months ago, there was a 100+ post thread on this mailing list complaining about long instance startup times (specifically, that classloading was slow). You guys had an internal discussion, and produced some code to streamline

Re: [google-appengine] Re: APP DOWN: App suddenly no longer starts, no code changes

2012-09-13 Thread Christina Ilvento
Hi All, First, thanks for all of the feedback, it's very helpful for us to hear directly from customers about how issues affect them and what would make them feel more confident using the platform. Please feel free to reach out to me directly (cilvento@) if you have anything that you'd like to

Re: [google-appengine] Re: APP DOWN: App suddenly no longer starts, no code changes

2012-09-13 Thread psm
Jeff, these are good ideas and suggestions. we are working on a number of different strategies to ameliorate these issues. some of the items you are suggesting are already in progress, and others besides. and i agree that this is a general philosophical challenge with PaaS. on GAE we now

Re: [google-appengine] Re: APP DOWN: App suddenly no longer starts, no code changes

2012-09-13 Thread Per
Hi Peter, I'm sure it's extremely(!) hard to host this many applications reliably and at a sane cost. I won't try to make any suggestions about that. But what concerns me is the support situation on App Engine. It shouldn't be too hard to monitor a low-bandwidth forum like this, and to

Re: [google-appengine] Re: APP DOWN: App suddenly no longer starts, no code changes

2012-09-13 Thread Peter Magnusson
point taken. i agree. On Thu, Sep 13, 2012 at 12:11 PM, Per per.fragem...@gmail.com wrote: Hi Peter, I'm sure it's extremely(!) hard to host this many applications reliably and at a sane cost. I won't try to make any suggestions about that. But what concerns me is the support situation

[google-appengine] Re: APP DOWN: App suddenly no longer starts, no code changes

2012-09-12 Thread Jeff Schnitzer
HELP! We have tried everything at this point. Shut down instances, tried to deploy a new version, even tried old versions. We've reported a production issue. Something is broken inside of GAE. The Guice error must be a symptom; the smoking gun seems to be:

[google-appengine] Re: APP DOWN: App suddenly no longer starts, no code changes

2012-09-12 Thread Jeff Schnitzer
More information: * The failure began at 10:54am (pacific). * Same app on different appid has the same problem. (as listed in the stacktraces, the appid is voost0) Jeff On Wed, Sep 12, 2012 at 11:53 AM, Jeff Schnitzer j...@infohazard.org wrote: HELP! We have tried everything

[google-appengine] Re: APP DOWN: App suddenly no longer starts, no code changes

2012-09-12 Thread Jeff Schnitzer
We are back up and running now after 2 hrs of downtime. To whoever fixed it: THANK YOU!!! To whoever broke it in the first place: SPANKINGS!!! Jeff On Wed, Sep 12, 2012 at 12:11 PM, Jeff Schnitzer j...@infohazard.org wrote: More information: * The failure began at 10:54am (pacific). *

[google-appengine] Re: APP DOWN: App suddenly no longer starts, no code changes

2012-09-12 Thread Kaan Soral
I wonder what happened, subscribed to this topic to be updated, hope someone explains what happened and additionally hope this never happens to me (python) :) On Wednesday, September 12, 2012 11:00:49 PM UTC+3, Jeff Schnitzer wrote: We are back up and running now after 2 hrs of downtime.

Re: [google-appengine] Re: APP DOWN: App suddenly no longer starts, no code changes

2012-09-12 Thread Christina Ilvento
*Hi All, Beginning yesterday, September 11, Google App Engine experienced two periods of serving degradation for a subset of Java applications due to a gradual roll-out of a new version of the Java runtime. Affected applications would have seen errors related to class loading. We have resolved

Re: [google-appengine] Re: APP DOWN: App suddenly no longer starts, no code changes

2012-09-12 Thread Kaan Soral
This is why I love App Engine, when a problem occurs instead of having a heart attack or committing suicide, you can just wait for it to be resolved. I remember downtime's that lasted nearly a day when I was on a custom server/php platform. App Engine really is a breeze, at the worst case this

Re: [google-appengine] Re: APP DOWN: App suddenly no longer starts, no code changes

2012-09-12 Thread Jon Stevens
I don't appreciate Google taking down my entire company for 2 hours because they are doing testing in production. We had one customer send out 500 emails today saying hey, check out this site and all they got was an error page. Really poor timing. Last time Google took my entire company down, it

Re: [google-appengine] Re: APP DOWN: App suddenly no longer starts, no code changes

2012-09-12 Thread Jeff Schnitzer
On Wed, Sep 12, 2012 at 3:12 PM, Kaan Soral kaanso...@gmail.com wrote: This is why I love App Engine, when a problem occurs instead of having a heart attack or committing suicide, you can just wait for it to be resolved. Hmmm. This really unfortunately timed incident may have cost us an

Re: [google-appengine] Re: APP DOWN: App suddenly no longer starts, no code changes

2012-09-12 Thread Thomas Wiradikusuma
Hi Jeff, I feel sorry for your loss. I agree completely with your message and your recommendation to Google. Hi GAE team, at the very least, please let us know when you want to roll out upgrades and stuff. It might not break stuff, but we need to know so we can be prepared (at least we won't