Hi Takashi, Here is the last request of one of these spikes, just search backwards from there.
59.167.133.100 - - [03/Oct/2012:16:56:02 -0700] "GET / HTTP/1.1" 500 0 "http://small-improvements.com/"; "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_7) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/22.0.1229.79 Safari/537.4" "www.small-improvements.com" ms=2463 cpu_ms=2210 exit_code=103 instance=00c61b117ca2c5dd4a6002dd13f39c6296da66 <https://appengine.google.com/instances?app_id=s%7Esmall-improvements-hrd&version_id=1.362133182361985490&key=00c61b117ca2c5dd4a6002dd13f39c6296da66#00c61b117ca2c5dd4a6002dd13f39c6296da66> Also, you have my application ID ("small-improvements-hrd"), and I gave you the general time. The log file is full of these errors, as you can see from the screenshot spikes. I have no idea why you wouldn't be able to find these? Does your search tool really yield "no errors"? Then it must be fundamentally broken. If I run a search I get hundreds of stacktraces. Of course I also raised a production error too, which has the exact time. I would have sent you a bug tracker ID for my production issue, if your bug tracker had such a thing. Every support system I've worked with in the past 10 years sends IDs in an email receipt. Yours does not. It seems to be GAE policy to keep customers in the dark, and this makes it a lot harder to tell you about issues -- but you do have my application ID and my name, so it shouldn't be hard to retrieve it that way either. Cheers, Per On Thursday, October 4, 2012 9:22:39 AM UTC+2, Takashi Matsuo (Google) wrote: > > Hi Per, > > Sorry, but I'm not able to locate the log you pasted. Can you confirm the > actual time(or time period) of the occurrence? > Do you still see the issue? If yes, can you file a production issue on our > issue tracker with more details? > > -- Takashi > > > On Thu, Oct 4, 2012 at 5:07 AM, Per <per.fr...@gmail.com <javascript:>>wrote: > >> >> <https://lh4.googleusercontent.com/-CQmIN9Qm_cY/UGyaOGMP9iI/AAAAAAAAAEU/zeYKUS7dxkc/s1600/Dashboard+-+Small+Improvements-1.jpg> >> >> >> I only realised after posting the other thread (about the response time >> increase) that we're actually encountering hundreds and hundreds of database >> errors. >> >> As always, the log file doesn't tell me anything. It's the generic >> DATASTORE_V3.Get() all over. We didn't make any deployment, we didnd't >> change any code recently, the load pattern >> >> didn't change, and there is no common trend either. The errors occur for an >> hour or so, strike everywhere (dozens of different clients are affected) and >> then the problem disappears >> for a few hours. >> >> Here's the stacktrace again (from about 90 minutes ago): >> >> https://www.small-improvements.com/dashboard >> javax.servlet.ServletException: >> com.google.apphosting.api.ApiProxy$ArgumentException: An error occurred >> parsing (locally or remotely) the arguments to DATASTORE_V3.Get(). >> at >> com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:268) >> at >> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) >> at org.mortbay.jetty.Server.handle(Server.java:326) >> at >> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) >> at >> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923) >> at >> com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java:76) >> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) >> at >> com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:146) >> at >> com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:447) >> at >> com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:452) >> at >> com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:459) >> at com.google.tracing.TraceContext.runInContext(TraceContext.java:701) >> at >> com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:336) >> at >> com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:328) >> at >> com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:456) >> at >> com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:251) >> at java.lang.Thread.run(Thread.java:679) >> Caused by: com.google.apphosting.api.ApiProxy$ArgumentException: An error >> occurred parsing (locally or remotely) the arguments to DATASTORE_V3.Get(). >> at >> com.google.apphosting.runtime.ApiProxyImpl$AsyncApiFuture.success(ApiProxyImpl.java:464) >> at >> com.google.apphosting.runtime.ApiProxyImpl$AsyncApiFuture.success(ApiProxyImpl.java:392) >> at >> com.google.net.rpc3.client.RpcStub$RpcCallbackDispatcher$1.runInContext(RpcStub.java:781) >> at >> com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:459) >> at com.google.tracing.TraceContext.runInContext(TraceContext.java:701) >> at >> com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:336) >> at >> com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:328) >> at >> com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:456) >> at >> com.google.net.rpc3.client.RpcStub$RpcCallbackDispatcher.rpcFinished(RpcStub.java:823) >> at >> com.google.net.rpc3.client.RpcStub$RpcCallbackDispatcher.success(RpcStub.java:808) >> at >> com.google.net.rpc3.impl.client.RpcClientInternalContext.runCallbacks(RpcClientInternalContext.java:893) >> at >> com.google.net.rpc3.impl.client.RpcClientInternalContext.finishRpcAndNotifyApp(RpcClientInternalContext.java:798) >> at >> com.google.net.rpc3.impl.client.RpcNetChannel.afterFinishingActiveRpc(RpcNetChannel.java:1063) >> at >> com.google.net.rpc3.impl.client.RpcNetChannel.finishRpc(RpcNetChannel.java:911) >> at >> com.google.net.rpc3.impl.client.RpcNetChannel.handleResponse(RpcNetChannel.java:2265) >> at >> com.google.net.rpc3.impl.client.RpcNetChannel.messageReceived(RpcNetChannel.java:2068) >> at >> com.google.net.rpc3.impl.client.RpcNetChannel.access$2000(RpcNetChannel.java:143) >> at >> com.google.net.rpc3.impl.client.RpcNetChannel$TransportCallback.receivedMessage(RpcNetChannel.java:3127) >> at >> com.google.net.rpc3.impl.client.RpcChannelTransportData$TransportCallback.receivedMessage(RpcChannelTransportData.java:599) >> at >> com.google.net.rpc3.impl.wire.RpcBaseTransport.receivedMessage(RpcBaseTransport.java:417) >> at >> com.google.apphosting.runtime.udrpc.UdrpcTransport$ClientAdapter.receivedMessage(UdrpcTransport.java:424) >> at >> com.google.apphosting.runtime.udrpc.UdrpcTransport.dispatchPacket(UdrpcTransport.java:265) >> at >> com.google.apphosting.runtime.udrpc.UdrpcTransport.readPackets(UdrpcTransport.java:217) >> at >> com.google.apphosting.runtime.udrpc.UdrpcTransport$1.run(UdrpcTransport.java:81) >> at >> com.google.net.eventmanager.AbstractFutureTask$Sync.innerRun(AbstractFutureTask.java:260) >> at >> com.google.net.eventmanager.AbstractFutureTask.run(AbstractFutureTask.java:121) >> at >> com.google.net.eventmanager.EventManagerImpl.runTask(EventManagerImpl.java:578) >> at >> com.google.net.eventmanager.EventManagerImpl.internalRunWorkerLoop(EventManagerImpl.java:1002) >> at >> com.google.net.eventmanager.EventManagerImpl.runWorkerLoop(EventManagerImpl.java:884) >> at >> com.google.net.eventmanager.WorkerThreadInfo.runWorkerLoop(WorkerThreadInfo.java:136) >> at >> com.google.net.eventmanager.EventManagerImpl$WorkerThread.run(EventManagerImpl.java:1855) >> >> >> And this is the error spike distribution today: >> >> >> >> <https://lh4.googleusercontent.com/-CQmIN9Qm_cY/UGyaOGMP9iI/AAAAAAAAAEU/zeYKUS7dxkc/s1600/Dashboard+-+Small+Improvements-1.jpg> >> >> >> I find it very frustrating that GAE is so inconsistent. We had severe issues >> two months ago and had to reimburse several clients for downtimes caused by >> very similar problems. Then it was fine for 2 months, >> and just when I go on vacation, it breaks down again. Anything I can do on >> my end? >> >> >> Cheers, >> Per >> >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To view this discussion on the web visit >> https://groups.google.com/d/msg/google-appengine/-/U9KKQ-WMZ98J. >> To post to this group, send email to >> google-a...@googlegroups.com<javascript:> >> . >> To unsubscribe from this group, send email to >> google-appengi...@googlegroups.com <javascript:>. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> > > > > -- > Takashi Matsuo | Developers Advocate | tma...@google.com <javascript:> > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/zwEIyb-Sf6oJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
