Hi Per, I have informed this issue to the engineering team. Thanks for the report. Anyway, it seems that now the error stopped. Can you confirm?
-- Takashi On Thu, Oct 4, 2012 at 7:34 PM, Takashi Matsuo <tmat...@google.com> wrote: > > Hi Per, > > I couldn't find the logs because I searched for 'Error' severity. Sorry > about that. Now I can see your warning logs, escalating it to the > engineering team. > > If you want to have a tracker ID, please use our issue tracker at: > http://code.google.com/p/googleappengine/issues/list > > As you say, the 'Report production issue' feature in the admin console > lacks a tracking feature, so we're re-thinking about this tool. > > -- Takashi > > > On Thu, Oct 4, 2012 at 6:41 PM, Per <per.fragem...@gmail.com> wrote: > >> >> Hi Takashi, >> >> Here is the last request of one of these spikes, just search backwards >> from there. >> >> 59.167.133.100 - - [03/Oct/2012:16:56:02 -0700] "GET / HTTP/1.1" 500 0 >> "http://small-improvements.com/"; "Mozilla/5.0 >> >> (Macintosh; Intel Mac OS X 10_6_7) AppleWebKit/537.4 (KHTML, like Gecko) >> Chrome/22.0.1229.79 Safari/537.4" >> "www.small-improvements.com" ms=2463 cpu_ms=2210 exit_code=103 >> instance=00c61b117ca2c5dd4a6002dd13f39c6296da66 >> <https://appengine.google.com/instances?app_id=s%7Esmall-improvements-hrd&version_id=1.362133182361985490&key=00c61b117ca2c5dd4a6002dd13f39c6296da66#00c61b117ca2c5dd4a6002dd13f39c6296da66> >> >> >> Also, you have my application ID ("small-improvements-hrd"), and I gave >> you the general time. The log file is full of these errors, as you can see >> from the screenshot spikes. I have no idea why you wouldn't be able to find >> these? Does your search tool really yield "no errors"? Then it must be >> fundamentally broken. If I run a search I get hundreds of stacktraces. >> >> Of course I also raised a production error too, which has the exact time. >> I would have sent you a bug tracker ID for my production issue, if your bug >> tracker had such a thing. Every support system I've worked with in the past >> 10 years sends IDs in an email receipt. Yours does not. It seems to be GAE >> policy to keep customers in the dark, and this makes it a lot harder to >> tell you about issues -- but you do have my application ID and my name, so >> it shouldn't be hard to retrieve it that way either. >> >> Cheers, >> Per >> >> >> >> >> >> On Thursday, October 4, 2012 9:22:39 AM UTC+2, Takashi Matsuo (Google) >> wrote: >> >>> Hi Per, >>> >>> Sorry, but I'm not able to locate the log you pasted. Can you confirm >>> the actual time(or time period) of the occurrence? >>> Do you still see the issue? If yes, can you file a production issue on >>> our issue tracker with more details? >>> >>> -- Takashi >>> >>> >>> On Thu, Oct 4, 2012 at 5:07 AM, Per <per.fr...@gmail.com> wrote: >>> >>>> >>>> <https://lh4.googleusercontent.com/-CQmIN9Qm_cY/UGyaOGMP9iI/AAAAAAAAAEU/zeYKUS7dxkc/s1600/Dashboard+-+Small+Improvements-1.jpg> >>>> >>>> >>>> I only realised after posting the other thread (about the response time >>>> increase) that we're actually encountering hundreds and hundreds of >>>> database errors. >>>> >>>> As always, the log file doesn't tell me anything. It's the generic >>>> DATASTORE_V3.Get() all over. We didn't make any deployment, we didnd't >>>> change any code recently, the load pattern >>>> >>>> >>>> didn't change, and there is no common trend either. The errors occur for >>>> an hour or so, strike everywhere (dozens of different clients are >>>> affected) and then the problem disappears >>>> for a few hours. >>>> >>>> Here's the stacktrace again (from about 90 minutes ago): >>>> >>>> >>>> https://www.small-**improvements.com/dashboard >>>> <https://www.small-improvements.com/dashboard> >>>> javax.servlet.**ServletException: >>>> com.google.apphosting.api.**ApiProxy$ArgumentException: An error occurred >>>> parsing (locally or remotely) the arguments to DATASTORE_V3.Get(). >>>> at >>>> com.google.apphosting.runtime.**jetty.AppVersionHandlerMap.**handle(AppVersionHandlerMap.**java:268) >>>> at >>>> org.mortbay.jetty.handler.**HandlerWrapper.handle(**HandlerWrapper.java:152) >>>> at org.mortbay.jetty.Server.**handle(Server.java:326) >>>> at >>>> org.mortbay.jetty.**HttpConnection.handleRequest(**HttpConnection.java:542) >>>> at >>>> org.mortbay.jetty.**HttpConnection$RequestHandler.**headerComplete(HttpConnection.**java:923) >>>> at >>>> com.google.apphosting.runtime.**jetty.RpcRequestParser.**parseAvailable(**RpcRequestParser.java:76) >>>> at org.mortbay.jetty.**HttpConnection.handle(**HttpConnection.java:404) >>>> at >>>> com.google.apphosting.runtime.**jetty.**JettyServletEngineAdapter.**serviceRequest(**JettyServletEngineAdapter.**java:146) >>>> at >>>> com.google.apphosting.runtime.**JavaRuntime$RequestRunnable.**run(JavaRuntime.java:447) >>>> at >>>> com.google.tracing.**TraceContext$**TraceContextRunnable.**runInContext(TraceContext.**java:452) >>>> at >>>> com.google.tracing.**TraceContext$**TraceContextRunnable$1.run(**TraceContext.java:459) >>>> at >>>> com.google.tracing.**TraceContext.runInContext(**TraceContext.java:701) >>>> at >>>> com.google.tracing.**TraceContext$**AbstractTraceContextCallback.**runInInheritedContextNoUnref(**TraceContext.java:336) >>>> at >>>> com.google.tracing.**TraceContext$**AbstractTraceContextCallback.**runInInheritedContext(**TraceContext.java:328) >>>> at >>>> com.google.tracing.**TraceContext$**TraceContextRunnable.run(**TraceContext.java:456) >>>> at >>>> com.google.apphosting.runtime.**ThreadGroupPool$PoolEntry.run(**ThreadGroupPool.java:251) >>>> at java.lang.Thread.run(Thread.**java:679) >>>> Caused by: com.google.apphosting.api.**ApiProxy$ArgumentException: An >>>> error occurred parsing (locally or remotely) the arguments to >>>> DATASTORE_V3.Get(). >>>> at >>>> com.google.apphosting.runtime.**ApiProxyImpl$AsyncApiFuture.**success(ApiProxyImpl.java:464) >>>> at >>>> com.google.apphosting.runtime.**ApiProxyImpl$AsyncApiFuture.**success(ApiProxyImpl.java:392) >>>> at >>>> com.google.net.rpc3.client.**RpcStub$RpcCallbackDispatcher$**1.runInContext(RpcStub.java:**781) >>>> at >>>> com.google.tracing.**TraceContext$**TraceContextRunnable$1.run(**TraceContext.java:459) >>>> at >>>> com.google.tracing.**TraceContext.runInContext(**TraceContext.java:701) >>>> at >>>> com.google.tracing.**TraceContext$**AbstractTraceContextCallback.**runInInheritedContextNoUnref(**TraceContext.java:336) >>>> at >>>> com.google.tracing.**TraceContext$**AbstractTraceContextCallback.**runInInheritedContext(**TraceContext.java:328) >>>> at >>>> com.google.tracing.**TraceContext$**TraceContextRunnable.run(**TraceContext.java:456) >>>> at >>>> com.google.net.rpc3.client.**RpcStub$RpcCallbackDispatcher.**rpcFinished(RpcStub.java:823) >>>> at >>>> com.google.net.rpc3.client.**RpcStub$RpcCallbackDispatcher.**success(RpcStub.java:808) >>>> at >>>> com.google.net.rpc3.impl.**client.**RpcClientInternalContext.**runCallbacks(**RpcClientInternalContext.java:**893) >>>> at >>>> com.google.net.rpc3.impl.**client.**RpcClientInternalContext.**finishRpcAndNotifyApp(**RpcClientInternalContext.java:**798) >>>> at >>>> com.google.net.rpc3.impl.**client.RpcNetChannel.**afterFinishingActiveRpc(**RpcNetChannel.java:1063) >>>> at >>>> com.google.net.rpc3.impl.**client.RpcNetChannel.**finishRpc(RpcNetChannel.java:**911) >>>> at >>>> com.google.net.rpc3.impl.**client.RpcNetChannel.**handleResponse(RpcNetChannel.**java:2265) >>>> at >>>> com.google.net.rpc3.impl.**client.RpcNetChannel.**messageReceived(RpcNetChannel.**java:2068) >>>> at >>>> com.google.net.rpc3.impl.**client.RpcNetChannel.access$**2000(RpcNetChannel.java:143) >>>> at >>>> com.google.net.rpc3.impl.**client.RpcNetChannel$**TransportCallback.**receivedMessage(RpcNetChannel.**java:3127) >>>> at >>>> com.google.net.rpc3.impl.**client.**RpcChannelTransportData$**TransportCallback.**receivedMessage(**RpcChannelTransportData.java:**599) >>>> at >>>> com.google.net.rpc3.impl.wire.**RpcBaseTransport.**receivedMessage(**RpcBaseTransport.java:417) >>>> at >>>> com.google.apphosting.runtime.**udrpc.UdrpcTransport$**ClientAdapter.receivedMessage(**UdrpcTransport.java:424) >>>> at >>>> com.google.apphosting.runtime.**udrpc.UdrpcTransport.**dispatchPacket(UdrpcTransport.**java:265) >>>> at >>>> com.google.apphosting.runtime.**udrpc.UdrpcTransport.**readPackets(UdrpcTransport.**java:217) >>>> at >>>> com.google.apphosting.runtime.**udrpc.UdrpcTransport$1.run(**UdrpcTransport.java:81) >>>> at >>>> com.google.net.eventmanager.**AbstractFutureTask$Sync.**innerRun(AbstractFutureTask.**java:260) >>>> at >>>> com.google.net.eventmanager.**AbstractFutureTask.run(**AbstractFutureTask.java:121) >>>> at >>>> com.google.net.eventmanager.**EventManagerImpl.runTask(**EventManagerImpl.java:578) >>>> at >>>> com.google.net.eventmanager.**EventManagerImpl.**internalRunWorkerLoop(**EventManagerImpl.java:1002) >>>> at >>>> com.google.net.eventmanager.**EventManagerImpl.**runWorkerLoop(**EventManagerImpl.java:884) >>>> at >>>> com.google.net.eventmanager.**WorkerThreadInfo.**runWorkerLoop(**WorkerThreadInfo.java:136) >>>> at >>>> com.google.net.eventmanager.**EventManagerImpl$WorkerThread.**run(EventManagerImpl.java:**1855) >>>> >>>> >>>> And this is the error spike distribution today: >>>> >>>> >>>> >>>> <https://lh4.googleusercontent.com/-CQmIN9Qm_cY/UGyaOGMP9iI/AAAAAAAAAEU/zeYKUS7dxkc/s1600/Dashboard+-+Small+Improvements-1.jpg> >>>> >>>> >>>> >>>> I find it very frustrating that GAE is so inconsistent. We had severe >>>> issues two months ago and had to reimburse several clients for downtimes >>>> caused by very similar problems. Then it was fine for 2 months, >>>> and just when I go on vacation, it breaks down again. Anything I can do on >>>> my end? >>>> >>>> >>>> >>>> Cheers, >>>> Per >>>> >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Google App Engine" group. >>>> To view this discussion on the web visit https://groups.google.com/d/** >>>> msg/google-appengine/-/U9KKQ-**WMZ98J<https://groups.google.com/d/msg/google-appengine/-/U9KKQ-WMZ98J> >>>> . >>>> To post to this group, send email to google-a...@googlegroups.**com. >>>> To unsubscribe from this group, send email to google-appengi...@** >>>> googlegroups.com. >>>> >>>> For more options, visit this group at http://groups.google.com/** >>>> group/google-appengine?hl=en<http://groups.google.com/group/google-appengine?hl=en> >>>> . >>>> >>> >>> >>> >>> -- >>> Takashi Matsuo | Developers Advocate | tma...@google.com >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To view this discussion on the web visit >> https://groups.google.com/d/msg/google-appengine/-/zwEIyb-Sf6oJ. >> >> To post to this group, send email to google-appengine@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> > > > > -- > Takashi Matsuo | Developers Advocate | tmat...@google.com > > -- Takashi Matsuo | Developers Advocate | tmat...@google.com -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
