<https://lh4.googleusercontent.com/-CQmIN9Qm_cY/UGyaOGMP9iI/AAAAAAAAAEU/zeYKUS7dxkc/s1600/Dashboard+-+Small+Improvements-1.jpg>
I only realised after posting the other thread (about the response time increase) that we're actually encountering hundreds and hundreds of database errors. As always, the log file doesn't tell me anything. It's the generic DATASTORE_V3.Get() all over. We didn't make any deployment, we didnd't change any code recently, the load pattern didn't change, and there is no common trend either. The errors occur for an hour or so, strike everywhere (dozens of different clients are affected) and then the problem disappears for a few hours. Here's the stacktrace again (from about 90 minutes ago): https://www.small-improvements.com/dashboard javax.servlet.ServletException: com.google.apphosting.api.ApiProxy$ArgumentException: An error occurred parsing (locally or remotely) the arguments to DATASTORE_V3.Get(). at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:268) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923) at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java:76) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:146) at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:447) at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:452) at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:459) at com.google.tracing.TraceContext.runInContext(TraceContext.java:701) at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:336) at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:328) at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:456) at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:251) at java.lang.Thread.run(Thread.java:679) Caused by: com.google.apphosting.api.ApiProxy$ArgumentException: An error occurred parsing (locally or remotely) the arguments to DATASTORE_V3.Get(). at com.google.apphosting.runtime.ApiProxyImpl$AsyncApiFuture.success(ApiProxyImpl.java:464) at com.google.apphosting.runtime.ApiProxyImpl$AsyncApiFuture.success(ApiProxyImpl.java:392) at com.google.net.rpc3.client.RpcStub$RpcCallbackDispatcher$1.runInContext(RpcStub.java:781) at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:459) at com.google.tracing.TraceContext.runInContext(TraceContext.java:701) at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:336) at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:328) at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:456) at com.google.net.rpc3.client.RpcStub$RpcCallbackDispatcher.rpcFinished(RpcStub.java:823) at com.google.net.rpc3.client.RpcStub$RpcCallbackDispatcher.success(RpcStub.java:808) at com.google.net.rpc3.impl.client.RpcClientInternalContext.runCallbacks(RpcClientInternalContext.java:893) at com.google.net.rpc3.impl.client.RpcClientInternalContext.finishRpcAndNotifyApp(RpcClientInternalContext.java:798) at com.google.net.rpc3.impl.client.RpcNetChannel.afterFinishingActiveRpc(RpcNetChannel.java:1063) at com.google.net.rpc3.impl.client.RpcNetChannel.finishRpc(RpcNetChannel.java:911) at com.google.net.rpc3.impl.client.RpcNetChannel.handleResponse(RpcNetChannel.java:2265) at com.google.net.rpc3.impl.client.RpcNetChannel.messageReceived(RpcNetChannel.java:2068) at com.google.net.rpc3.impl.client.RpcNetChannel.access$2000(RpcNetChannel.java:143) at com.google.net.rpc3.impl.client.RpcNetChannel$TransportCallback.receivedMessage(RpcNetChannel.java:3127) at com.google.net.rpc3.impl.client.RpcChannelTransportData$TransportCallback.receivedMessage(RpcChannelTransportData.java:599) at com.google.net.rpc3.impl.wire.RpcBaseTransport.receivedMessage(RpcBaseTransport.java:417) at com.google.apphosting.runtime.udrpc.UdrpcTransport$ClientAdapter.receivedMessage(UdrpcTransport.java:424) at com.google.apphosting.runtime.udrpc.UdrpcTransport.dispatchPacket(UdrpcTransport.java:265) at com.google.apphosting.runtime.udrpc.UdrpcTransport.readPackets(UdrpcTransport.java:217) at com.google.apphosting.runtime.udrpc.UdrpcTransport$1.run(UdrpcTransport.java:81) at com.google.net.eventmanager.AbstractFutureTask$Sync.innerRun(AbstractFutureTask.java:260) at com.google.net.eventmanager.AbstractFutureTask.run(AbstractFutureTask.java:121) at com.google.net.eventmanager.EventManagerImpl.runTask(EventManagerImpl.java:578) at com.google.net.eventmanager.EventManagerImpl.internalRunWorkerLoop(EventManagerImpl.java:1002) at com.google.net.eventmanager.EventManagerImpl.runWorkerLoop(EventManagerImpl.java:884) at com.google.net.eventmanager.WorkerThreadInfo.runWorkerLoop(WorkerThreadInfo.java:136) at com.google.net.eventmanager.EventManagerImpl$WorkerThread.run(EventManagerImpl.java:1855) And this is the error spike distribution today: <https://lh4.googleusercontent.com/-CQmIN9Qm_cY/UGyaOGMP9iI/AAAAAAAAAEU/zeYKUS7dxkc/s1600/Dashboard+-+Small+Improvements-1.jpg> I find it very frustrating that GAE is so inconsistent. We had severe issues two months ago and had to reimburse several clients for downtimes caused by very similar problems. Then it was fine for 2 months, and just when I go on vacation, it breaks down again. Anything I can do on my end? Cheers, Per -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/U9KKQ-WMZ98J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
