Hello Simon, This is the callstack on exception we were getting in the JSPs, which are not using anything memcache related:
org.apache.jasper.runtime.JspFactoryImpl internalGetPageContext: Exception initializing page context com.google.appengine.api.memcache.MemcacheServiceException: Memcache put: Error setting single item (_ahsSqsA9jzIMmQjzHzlJI1ozQ) at com.google.appengine.api.memcache.AsyncMemcacheServiceImpl $5.transform(AsyncMemcacheServiceImpl.java:380) at com.google.appengine.api.memcache.AsyncMemcacheServiceImpl $5.transform(AsyncMemcacheServiceImpl.java:372) at com.google.appengine.api.memcache.MemcacheServiceApiHelper $RpcResponseHandler.convertResponse(MemcacheServiceApiHelper.java:59) at com.google.appengine.api.memcache.MemcacheServiceApiHelper $1.wrap(MemcacheServiceApiHelper.java:98) at com.google.appengine.api.memcache.MemcacheServiceApiHelper $1.wrap(MemcacheServiceApiHelper.java:92) at com.google.appengine.api.utils.FutureWrapper.wrapAndCache(FutureWrapper.java: 58) at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java: 78) at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java: 74) at com.google.appengine.api.memcache.MemcacheServiceImpl.quietGet(MemcacheServiceImpl.java: 27) at com.google.appengine.api.memcache.MemcacheServiceImpl.put(MemcacheServiceImpl.java: 74) at com.google.apphosting.runtime.jetty.MemcacheSessionStore.saveSession(MemcacheSessionStore.java: 39) at com.google.apphosting.runtime.jetty.SessionManager.createSession(SessionManager.java: 342) at com.google.apphosting.runtime.jetty.SessionManager $AppEngineSession.<init>(SessionManager.java:125) at com.google.apphosting.runtime.jetty.SessionManager.newSession(SessionManager.java: 273) at com.google.apphosting.runtime.jetty.SessionManager.newSession(SessionManager.java: 43) at org.mortbay.jetty.servlet.AbstractSessionManager.newHttpSession(AbstractSessionManager.java: 413) at org.mortbay.jetty.Request.getSession(Request.java:1242) at org.mortbay.jetty.Request.getSession(Request.java:1214) at org.apache.jasper.runtime.PageContextImpl._initialize(PageContextImpl.java: 146) at org.apache.jasper.runtime.PageContextImpl.initialize(PageContextImpl.java: 124) at org.apache.jasper.runtime.JspFactoryImpl.internalGetPageContext(JspFactoryImpl.java: 107) at org.apache.jasper.runtime.JspFactoryImpl.access $000(JspFactoryImpl.java:40) at org.apache.jasper.runtime.JspFactoryImpl $PrivilegedGetPageContext.run(JspFactoryImpl.java:149) at com.google.apphosting.runtime.security.shared.intercept.java.security.AccessController_.doPrivileged(AccessController_.java: 34) at org.apache.jasper.runtime.JspFactoryImpl.getPageContext(JspFactoryImpl.java: 61) at org.apache.jsp.plans_jsp._jspService(plans_jsp.java:54) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java: 511) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1166) at com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlobUploadFilter.java: 97) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1157) at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionFilter.java: 35) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1157) at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java: 43) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1157) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java: 388) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java: 216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java: 182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java: 765) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java: 418) at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java: 249) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java: 152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java: 542) at org.mortbay.jetty.HttpConnection $RequestHandler.headerComplete(HttpConnection.java:923) at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java: 76) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java: 135) at com.google.apphosting.runtime.JavaRuntime $RequestRunnable.run(JavaRuntime.java:393) at com.google.tracing.TraceContext $TraceContextRunnable.runInContext(TraceContext.java:449) at com.google.tracing.TraceContext$TraceContextRunnable $1.run(TraceContext.java:455) at com.google.tracing.TraceContext.runInContext(TraceContext.java:695) at com.google.tracing.TraceContext $AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java: 333) at com.google.tracing.TraceContext $AbstractTraceContextCallback.runInInheritedContext(TraceContext.java: 325) at com.google.tracing.TraceContext $TraceContextRunnable.run(TraceContext.java:453) at com.google.apphosting.runtime.ThreadGroupPool $PoolEntry.run(ThreadGroupPool.java:160) at java.lang.Thread.run(Thread.java:679) As you can see, the issue is in the session management is using memcache, and doing a put, which is what was exception'ing for 1 hour and 20 minutes. As discussed in this other thread http://groups.google.com/group/google-appengine/browse_thread/thread/200e6040039e14e1# this exception should have never happened, as the MemcacheService ErrorHandler default value is LogAndContinueErrorHandler. The GAE session management code counts on this as well, and was taken down by this outage. This outage was definitely localized to only some apps, but apps affected using session management were completely down. Still no word from Google about the outage, what happened, what is being done to ensure this chain failure does not happen again, etc. Jerome On Oct 26, 9:56 am, Simon Knott <knott.si...@gmail.com> wrote: > Can you post one of the exceptions which was thrown from your JSP pages? > That seems a little strange to me! -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
