Hi Per, I couldn't find the logs because I searched for 'Error' severity. Sorry about that. Now I can see your warning logs, escalating it to the engineering team.
If you want to have a tracker ID, please use our issue tracker at: http://code.google.com/p/googleappengine/issues/list As you say, the 'Report production issue' feature in the admin console lacks a tracking feature, so we're re-thinking about this tool. -- Takashi On Thu, Oct 4, 2012 at 6:41 PM, Per <per.fragem...@gmail.com> wrote: > > Hi Takashi, > > Here is the last request of one of these spikes, just search backwards > from there. > > 59.167.133.100 - - [03/Oct/2012:16:56:02 -0700] "GET / HTTP/1.1" 500 0 > "http://small-improvements.com/"; "Mozilla/5.0 > (Macintosh; Intel Mac OS X 10_6_7) AppleWebKit/537.4 (KHTML, like Gecko) > Chrome/22.0.1229.79 Safari/537.4" > "www.small-improvements.com" ms=2463 cpu_ms=2210 exit_code=103 > instance=00c61b117ca2c5dd4a6002dd13f39c6296da66 > <https://appengine.google.com/instances?app_id=s%7Esmall-improvements-hrd&version_id=1.362133182361985490&key=00c61b117ca2c5dd4a6002dd13f39c6296da66#00c61b117ca2c5dd4a6002dd13f39c6296da66> > > > Also, you have my application ID ("small-improvements-hrd"), and I gave > you the general time. The log file is full of these errors, as you can see > from the screenshot spikes. I have no idea why you wouldn't be able to find > these? Does your search tool really yield "no errors"? Then it must be > fundamentally broken. If I run a search I get hundreds of stacktraces. > > Of course I also raised a production error too, which has the exact time. > I would have sent you a bug tracker ID for my production issue, if your bug > tracker had such a thing. Every support system I've worked with in the past > 10 years sends IDs in an email receipt. Yours does not. It seems to be GAE > policy to keep customers in the dark, and this makes it a lot harder to > tell you about issues -- but you do have my application ID and my name, so > it shouldn't be hard to retrieve it that way either. > > Cheers, > Per > > > > > > On Thursday, October 4, 2012 9:22:39 AM UTC+2, Takashi Matsuo (Google) > wrote: > >> Hi Per, >> >> Sorry, but I'm not able to locate the log you pasted. Can you confirm the >> actual time(or time period) of the occurrence? >> Do you still see the issue? If yes, can you file a production issue on >> our issue tracker with more details? >> >> -- Takashi >> >> >> On Thu, Oct 4, 2012 at 5:07 AM, Per <per.fr...@gmail.com> wrote: >> >>> >>> <https://lh4.googleusercontent.com/-CQmIN9Qm_cY/UGyaOGMP9iI/AAAAAAAAAEU/zeYKUS7dxkc/s1600/Dashboard+-+Small+Improvements-1.jpg> >>> >>> >>> I only realised after posting the other thread (about the response time >>> increase) that we're actually encountering hundreds and hundreds of >>> database errors. >>> >>> As always, the log file doesn't tell me anything. It's the generic >>> DATASTORE_V3.Get() all over. We didn't make any deployment, we didnd't >>> change any code recently, the load pattern >>> >>> didn't change, and there is no common trend either. The errors occur for an >>> hour or so, strike everywhere (dozens of different clients are affected) >>> and then the problem disappears >>> for a few hours. >>> >>> Here's the stacktrace again (from about 90 minutes ago): >>> >>> https://www.small-**improvements.com/dashboard >>> <https://www.small-improvements.com/dashboard> >>> javax.servlet.**ServletException: >>> com.google.apphosting.api.**ApiProxy$ArgumentException: An error occurred >>> parsing (locally or remotely) the arguments to DATASTORE_V3.Get(). >>> at >>> com.google.apphosting.runtime.**jetty.AppVersionHandlerMap.**handle(AppVersionHandlerMap.**java:268) >>> at >>> org.mortbay.jetty.handler.**HandlerWrapper.handle(**HandlerWrapper.java:152) >>> at org.mortbay.jetty.Server.**handle(Server.java:326) >>> at >>> org.mortbay.jetty.**HttpConnection.handleRequest(**HttpConnection.java:542) >>> at >>> org.mortbay.jetty.**HttpConnection$RequestHandler.**headerComplete(HttpConnection.**java:923) >>> at >>> com.google.apphosting.runtime.**jetty.RpcRequestParser.**parseAvailable(**RpcRequestParser.java:76) >>> at org.mortbay.jetty.**HttpConnection.handle(**HttpConnection.java:404) >>> at >>> com.google.apphosting.runtime.**jetty.**JettyServletEngineAdapter.**serviceRequest(**JettyServletEngineAdapter.**java:146) >>> at >>> com.google.apphosting.runtime.**JavaRuntime$RequestRunnable.**run(JavaRuntime.java:447) >>> at >>> com.google.tracing.**TraceContext$**TraceContextRunnable.**runInContext(TraceContext.**java:452) >>> at >>> com.google.tracing.**TraceContext$**TraceContextRunnable$1.run(**TraceContext.java:459) >>> at >>> com.google.tracing.**TraceContext.runInContext(**TraceContext.java:701) >>> at >>> com.google.tracing.**TraceContext$**AbstractTraceContextCallback.**runInInheritedContextNoUnref(**TraceContext.java:336) >>> at >>> com.google.tracing.**TraceContext$**AbstractTraceContextCallback.**runInInheritedContext(**TraceContext.java:328) >>> at >>> com.google.tracing.**TraceContext$**TraceContextRunnable.run(**TraceContext.java:456) >>> at >>> com.google.apphosting.runtime.**ThreadGroupPool$PoolEntry.run(**ThreadGroupPool.java:251) >>> at java.lang.Thread.run(Thread.**java:679) >>> Caused by: com.google.apphosting.api.**ApiProxy$ArgumentException: An error >>> occurred parsing (locally or remotely) the arguments to DATASTORE_V3.Get(). >>> at >>> com.google.apphosting.runtime.**ApiProxyImpl$AsyncApiFuture.**success(ApiProxyImpl.java:464) >>> at >>> com.google.apphosting.runtime.**ApiProxyImpl$AsyncApiFuture.**success(ApiProxyImpl.java:392) >>> at >>> com.google.net.rpc3.client.**RpcStub$RpcCallbackDispatcher$**1.runInContext(RpcStub.java:**781) >>> at >>> com.google.tracing.**TraceContext$**TraceContextRunnable$1.run(**TraceContext.java:459) >>> at >>> com.google.tracing.**TraceContext.runInContext(**TraceContext.java:701) >>> at >>> com.google.tracing.**TraceContext$**AbstractTraceContextCallback.**runInInheritedContextNoUnref(**TraceContext.java:336) >>> at >>> com.google.tracing.**TraceContext$**AbstractTraceContextCallback.**runInInheritedContext(**TraceContext.java:328) >>> at >>> com.google.tracing.**TraceContext$**TraceContextRunnable.run(**TraceContext.java:456) >>> at >>> com.google.net.rpc3.client.**RpcStub$RpcCallbackDispatcher.**rpcFinished(RpcStub.java:823) >>> at >>> com.google.net.rpc3.client.**RpcStub$RpcCallbackDispatcher.**success(RpcStub.java:808) >>> at >>> com.google.net.rpc3.impl.**client.**RpcClientInternalContext.**runCallbacks(**RpcClientInternalContext.java:**893) >>> at >>> com.google.net.rpc3.impl.**client.**RpcClientInternalContext.**finishRpcAndNotifyApp(**RpcClientInternalContext.java:**798) >>> at >>> com.google.net.rpc3.impl.**client.RpcNetChannel.**afterFinishingActiveRpc(**RpcNetChannel.java:1063) >>> at >>> com.google.net.rpc3.impl.**client.RpcNetChannel.**finishRpc(RpcNetChannel.java:**911) >>> at >>> com.google.net.rpc3.impl.**client.RpcNetChannel.**handleResponse(RpcNetChannel.**java:2265) >>> at >>> com.google.net.rpc3.impl.**client.RpcNetChannel.**messageReceived(RpcNetChannel.**java:2068) >>> at >>> com.google.net.rpc3.impl.**client.RpcNetChannel.access$**2000(RpcNetChannel.java:143) >>> at >>> com.google.net.rpc3.impl.**client.RpcNetChannel$**TransportCallback.**receivedMessage(RpcNetChannel.**java:3127) >>> at >>> com.google.net.rpc3.impl.**client.**RpcChannelTransportData$**TransportCallback.**receivedMessage(**RpcChannelTransportData.java:**599) >>> at >>> com.google.net.rpc3.impl.wire.**RpcBaseTransport.**receivedMessage(**RpcBaseTransport.java:417) >>> at >>> com.google.apphosting.runtime.**udrpc.UdrpcTransport$**ClientAdapter.receivedMessage(**UdrpcTransport.java:424) >>> at >>> com.google.apphosting.runtime.**udrpc.UdrpcTransport.**dispatchPacket(UdrpcTransport.**java:265) >>> at >>> com.google.apphosting.runtime.**udrpc.UdrpcTransport.**readPackets(UdrpcTransport.**java:217) >>> at >>> com.google.apphosting.runtime.**udrpc.UdrpcTransport$1.run(**UdrpcTransport.java:81) >>> at >>> com.google.net.eventmanager.**AbstractFutureTask$Sync.**innerRun(AbstractFutureTask.**java:260) >>> at >>> com.google.net.eventmanager.**AbstractFutureTask.run(**AbstractFutureTask.java:121) >>> at >>> com.google.net.eventmanager.**EventManagerImpl.runTask(**EventManagerImpl.java:578) >>> at >>> com.google.net.eventmanager.**EventManagerImpl.**internalRunWorkerLoop(**EventManagerImpl.java:1002) >>> at >>> com.google.net.eventmanager.**EventManagerImpl.**runWorkerLoop(**EventManagerImpl.java:884) >>> at >>> com.google.net.eventmanager.**WorkerThreadInfo.**runWorkerLoop(**WorkerThreadInfo.java:136) >>> at >>> com.google.net.eventmanager.**EventManagerImpl$WorkerThread.**run(EventManagerImpl.java:**1855) >>> >>> >>> And this is the error spike distribution today: >>> >>> >>> >>> <https://lh4.googleusercontent.com/-CQmIN9Qm_cY/UGyaOGMP9iI/AAAAAAAAAEU/zeYKUS7dxkc/s1600/Dashboard+-+Small+Improvements-1.jpg> >>> >>> >>> I find it very frustrating that GAE is so inconsistent. We had severe >>> issues two months ago and had to reimburse several clients for downtimes >>> caused by very similar problems. Then it was fine for 2 months, >>> and just when I go on vacation, it breaks down again. Anything I can do on >>> my end? >>> >>> >>> Cheers, >>> Per >>> >>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Google App Engine" group. >>> To view this discussion on the web visit https://groups.google.com/d/** >>> msg/google-appengine/-/U9KKQ-**WMZ98J<https://groups.google.com/d/msg/google-appengine/-/U9KKQ-WMZ98J> >>> . >>> To post to this group, send email to google-a...@googlegroups.**com. >>> To unsubscribe from this group, send email to google-appengi...@** >>> googlegroups.com. >>> >>> For more options, visit this group at http://groups.google.com/** >>> group/google-appengine?hl=en<http://groups.google.com/group/google-appengine?hl=en> >>> . >>> >> >> >> >> -- >> Takashi Matsuo | Developers Advocate | tma...@google.com >> >> -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/zwEIyb-Sf6oJ. > > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- Takashi Matsuo | Developers Advocate | tmat...@google.com -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
