Hi Takashi, you're right, it seems to have stopped now. The response time is still a bit high, so I'm fearing the error could strike again, but at least right now nothing bad is happening.
I'm always searching using 'HTTP/1.1" 500' to identify failed requests, and I ignore the error level. Maybe I have to rework our logging in that regard, sorry if that confused you. I guess I'll stop using the "raise production error" feature then, I thought it was the preferred way of raising problems these days. Glad to hear it's not! And thanks for passing this on the engineering team :) Cheers, Per On Thursday, October 4, 2012 1:03:20 PM UTC+2, Takashi Matsuo (Google) wrote: > > > Hi Per, > > I have informed this issue to the engineering team. Thanks for the report. > Anyway, it seems that now the error stopped. Can you confirm? > > -- Takashi > > > > On Thu, Oct 4, 2012 at 7:34 PM, Takashi Matsuo <tma...@google.com<javascript:> > > wrote: > >> >> Hi Per, >> >> I couldn't find the logs because I searched for 'Error' severity. Sorry >> about that. Now I can see your warning logs, escalating it to the >> engineering team. >> >> If you want to have a tracker ID, please use our issue tracker at: >> http://code.google.com/p/googleappengine/issues/list >> >> As you say, the 'Report production issue' feature in the admin console >> lacks a tracking feature, so we're re-thinking about this tool. >> >> -- Takashi >> >> >> On Thu, Oct 4, 2012 at 6:41 PM, Per <per.fr...@gmail.com <javascript:>>wrote: >> >>> >>> Hi Takashi, >>> >>> Here is the last request of one of these spikes, just search backwards >>> from there. >>> >>> 59.167.133.100 - - [03/Oct/2012:16:56:02 -0700] "GET / HTTP/1.1" 500 0 >>> "http://small-improvements.com/"; "Mozilla/5.0 >>> >>> >>> (Macintosh; Intel Mac OS X 10_6_7) AppleWebKit/537.4 (KHTML, like Gecko) >>> Chrome/22.0.1229.79 Safari/537.4" >>> "www.small-improvements.com" ms=2463 cpu_ms=2210 exit_code=103 >>> instance=00c61b117ca2c5dd4a6002dd13f39c6296da66 >>> <https://appengine.google.com/instances?app_id=s%7Esmall-improvements-hrd&version_id=1.362133182361985490&key=00c61b117ca2c5dd4a6002dd13f39c6296da66#00c61b117ca2c5dd4a6002dd13f39c6296da66> >>> >>> >>> Also, you have my application ID ("small-improvements-hrd"), and I gave >>> you the general time. The log file is full of these errors, as you can see >>> from the screenshot spikes. I have no idea why you wouldn't be able to find >>> these? Does your search tool really yield "no errors"? Then it must be >>> fundamentally broken. If I run a search I get hundreds of stacktraces. >>> >>> Of course I also raised a production error too, which has the exact >>> time. I would have sent you a bug tracker ID for my production issue, if >>> your bug tracker had such a thing. Every support system I've worked with in >>> the past 10 years sends IDs in an email receipt. Yours does not. It seems >>> to be GAE policy to keep customers in the dark, and this makes it a lot >>> harder to tell you about issues -- but you do have my application ID and my >>> name, so it shouldn't be hard to retrieve it that way either. >>> >>> Cheers, >>> Per >>> >>> >>> >>> >>> >>> On Thursday, October 4, 2012 9:22:39 AM UTC+2, Takashi Matsuo (Google) >>> wrote: >>> >>>> Hi Per, >>>> >>>> Sorry, but I'm not able to locate the log you pasted. Can you confirm >>>> the actual time(or time period) of the occurrence? >>>> Do you still see the issue? If yes, can you file a production issue on >>>> our issue tracker with more details? >>>> >>>> -- Takashi >>>> >>>> >>>> On Thu, Oct 4, 2012 at 5:07 AM, Per <per.fr...@gmail.com> wrote: >>>> >>>>> >>>>> <https://lh4.googleusercontent.com/-CQmIN9Qm_cY/UGyaOGMP9iI/AAAAAAAAAEU/zeYKUS7dxkc/s1600/Dashboard+-+Small+Improvements-1.jpg> >>>>> >>>>> >>>>> I only realised after posting the other thread (about the response time >>>>> increase) that we're actually encountering hundreds and hundreds of >>>>> database errors. >>>>> >>>>> As always, the log file doesn't tell me anything. It's the generic >>>>> DATASTORE_V3.Get() all over. We didn't make any deployment, we didnd't >>>>> change any code recently, the load pattern >>>>> >>>>> >>>>> >>>>> didn't change, and there is no common trend either. The errors occur for >>>>> an hour or so, strike everywhere (dozens of different clients are >>>>> affected) and then the problem disappears >>>>> for a few hours. >>>>> >>>>> Here's the stacktrace again (from about 90 minutes ago): >>>>> >>>>> >>>>> >>>>> https://www.small-**improvements.com/dashboard >>>>> <https://www.small-improvements.com/dashboard> >>>>> javax.servlet.**ServletException: >>>>> com.google.apphosting.api.**ApiProxy$ArgumentException: An error occurred >>>>> parsing (locally or remotely) the arguments to DATASTORE_V3.Get(). >>>>> at >>>>> com.google.apphosting.runtime.**jetty.AppVersionHandlerMap.**handle(AppVersionHandlerMap.**java:268) >>>>> at >>>>> org.mortbay.jetty.handler.**HandlerWrapper.handle(**HandlerWrapper.java:152) >>>>> at org.mortbay.jetty.Server.**handle(Server.java:326) >>>>> at >>>>> org.mortbay.jetty.**HttpConnection.handleRequest(**HttpConnection.java:542) >>>>> at >>>>> org.mortbay.jetty.**HttpConnection$RequestHandler.**headerComplete(HttpConnection.**java:923) >>>>> at >>>>> com.google.apphosting.runtime.**jetty.RpcRequestParser.**parseAvailable(**RpcRequestParser.java:76) >>>>> at org.mortbay.jetty.**HttpConnection.handle(**HttpConnection.java:404) >>>>> at >>>>> com.google.apphosting.runtime.**jetty.**JettyServletEngineAdapter.**serviceRequest(**JettyServletEngineAdapter.**java:146) >>>>> at >>>>> com.google.apphosting.runtime.**JavaRuntime$RequestRunnable.**run(JavaRuntime.java:447) >>>>> at >>>>> com.google.tracing.**TraceContext$**TraceContextRunnable.**runInContext(TraceContext.**java:452) >>>>> at >>>>> com.google.tracing.**TraceContext$**TraceContextRunnable$1.run(**TraceContext.java:459) >>>>> at >>>>> com.google.tracing.**TraceContext.runInContext(**TraceContext.java:701) >>>>> at >>>>> com.google.tracing.**TraceContext$**AbstractTraceContextCallback.**runInInheritedContextNoUnref(**TraceContext.java:336) >>>>> at >>>>> com.google.tracing.**TraceContext$**AbstractTraceContextCallback.**runInInheritedContext(**TraceContext.java:328) >>>>> at >>>>> com.google.tracing.**TraceContext$**TraceContextRunnable.run(**TraceContext.java:456) >>>>> at >>>>> com.google.apphosting.runtime.**ThreadGroupPool$PoolEntry.run(**ThreadGroupPool.java:251) >>>>> at java.lang.Thread.run(Thread.**java:679) >>>>> Caused by: com.google.apphosting.api.**ApiProxy$ArgumentException: An >>>>> error occurred parsing (locally or remotely) the arguments to >>>>> DATASTORE_V3.Get(). >>>>> at >>>>> com.google.apphosting.runtime.**ApiProxyImpl$AsyncApiFuture.**success(ApiProxyImpl.java:464) >>>>> at >>>>> com.google.apphosting.runtime.**ApiProxyImpl$AsyncApiFuture.**success(ApiProxyImpl.java:392) >>>>> at >>>>> com.google.net.rpc3.client.**RpcStub$RpcCallbackDispatcher$**1.runInContext(RpcStub.java:**781) >>>>> at >>>>> com.google.tracing.**TraceContext$**TraceContextRunnable$1.run(**TraceContext.java:459) >>>>> at >>>>> com.google.tracing.**TraceContext.runInContext(**TraceContext.java:701) >>>>> at >>>>> com.google.tracing.**TraceContext$**AbstractTraceContextCallback.**runInInheritedContextNoUnref(**TraceContext.java:336) >>>>> at >>>>> com.google.tracing.**TraceContext$**AbstractTraceContextCallback.**runInInheritedContext(**TraceContext.java:328) >>>>> at >>>>> com.google.tracing.**TraceContext$**TraceContextRunnable.run(**TraceContext.java:456) >>>>> at >>>>> com.google.net.rpc3.client.**RpcStub$RpcCallbackDispatcher.**rpcFinished(RpcStub.java:823) >>>>> at >>>>> com.google.net.rpc3.client.**RpcStub$RpcCallbackDispatcher.**success(RpcStub.java:808) >>>>> at >>>>> com.google.net.rpc3.impl.**client.**RpcClientInternalContext.**runCallbacks(**RpcClientInternalContext.java:**893) >>>>> at >>>>> com.google.net.rpc3.impl.**client.**RpcClientInternalContext.**finishRpcAndNotifyApp(**RpcClientInternalContext.java:**798) >>>>> at >>>>> com.google.net.rpc3.impl.**client.RpcNetChannel.**afterFinishingActiveRpc(**RpcNetChannel.java:1063) >>>>> at >>>>> com.google.net.rpc3.impl.**client.RpcNetChannel.**finishRpc(RpcNetChannel.java:**911) >>>>> at >>>>> com.google.net.rpc3.impl.**client.RpcNetChannel.**handleResponse(RpcNetChannel.**java:2265) >>>>> at >>>>> com.google.net.rpc3.impl.**client.RpcNetChannel.**messageReceived(RpcNetChannel.**java:2068) >>>>> at >>>>> com.google.net.rpc3.impl.**client.RpcNetChannel.access$**2000(RpcNetChannel.java:143) >>>>> at >>>>> com.google.net.rpc3.impl.**client.RpcNetChannel$**TransportCallback.**receivedMessage(RpcNetChannel.**java:3127) >>>>> at >>>>> com.google.net.rpc3.impl.**client.**RpcChannelTransportData$**TransportCallback.**receivedMessage(**RpcChannelTransportData.java:**599) >>>>> at >>>>> com.google.net.rpc3.impl.wire.**RpcBaseTransport.**receivedMessage(**RpcBaseTransport.java:417) >>>>> at >>>>> com.google.apphosting.runtime.**udrpc.UdrpcTransport$**ClientAdapter.receivedMessage(**UdrpcTransport.java:424) >>>>> at >>>>> com.google.apphosting.runtime.**udrpc.UdrpcTransport.**dispatchPacket(UdrpcTransport.**java:265) >>>>> at >>>>> com.google.apphosting.runtime.**udrpc.UdrpcTransport.**readPackets(UdrpcTransport.**java:217) >>>>> at >>>>> com.google.apphosting.runtime.**udrpc.UdrpcTransport$1.run(**UdrpcTransport.java:81) >>>>> at >>>>> com.google.net.eventmanager.**AbstractFutureTask$Sync.**innerRun(AbstractFutureTask.**java:260) >>>>> at >>>>> com.google.net.eventmanager.**AbstractFutureTask.run(**AbstractFutureTask.java:121) >>>>> at >>>>> com.google.net.eventmanager.**EventManagerImpl.runTask(**EventManagerImpl.java:578) >>>>> at >>>>> com.google.net.eventmanager.**EventManagerImpl.**internalRunWorkerLoop(**EventManagerImpl.java:1002) >>>>> at >>>>> com.google.net.eventmanager.**EventManagerImpl.**runWorkerLoop(**EventManagerImpl.java:884) >>>>> at >>>>> com.google.net.eventmanager.**WorkerThreadInfo.**runWorkerLoop(**WorkerThreadInfo.java:136) >>>>> at >>>>> com.google.net.eventmanager.**EventManagerImpl$WorkerThread.**run(EventManagerImpl.java:**1855) >>>>> >>>>> >>>>> And this is the error spike distribution today: >>>>> >>>>> >>>>> >>>>> <https://lh4.googleusercontent.com/-CQmIN9Qm_cY/UGyaOGMP9iI/AAAAAAAAAEU/zeYKUS7dxkc/s1600/Dashboard+-+Small+Improvements-1.jpg> >>>>> >>>>> >>>>> >>>>> >>>>> I find it very frustrating that GAE is so inconsistent. We had severe >>>>> issues two months ago and had to reimburse several clients for downtimes >>>>> caused by very similar problems. Then it was fine for 2 months, >>>>> and just when I go on vacation, it breaks down again. Anything I can do >>>>> on my end? >>>>> >>>>> >>>>> >>>>> >>>>> Cheers, >>>>> Per >>>>> >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Google App Engine" group. >>>>> To view this discussion on the web visit https://groups.google.com/d/* >>>>> *msg/google-appengine/-/U9KKQ-**WMZ98J<https://groups.google.com/d/msg/google-appengine/-/U9KKQ-WMZ98J> >>>>> . >>>>> To post to this group, send email to google-a...@googlegroups.**com. >>>>> To unsubscribe from this group, send email to google-appengi...@** >>>>> googlegroups.com. >>>>> >>>>> For more options, visit this group at http://groups.google.com/** >>>>> group/google-appengine?hl=en<http://groups.google.com/group/google-appengine?hl=en> >>>>> . >>>>> >>>> >>>> >>>> >>>> -- >>>> Takashi Matsuo | Developers Advocate | tma...@google.com >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Google App Engine" group. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msg/google-appengine/-/zwEIyb-Sf6oJ. >>> >>> To post to this group, send email to >>> google-a...@googlegroups.com<javascript:> >>> . >>> To unsubscribe from this group, send email to >>> google-appengi...@googlegroups.com <javascript:>. >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine?hl=en. >>> >> >> >> >> -- >> Takashi Matsuo | Developers Advocate | tma...@google.com <javascript:> >> >> > > > -- > Takashi Matsuo | Developers Advocate | tma...@google.com <javascript:> > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/EYREqE9orl0J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
