Hi GAE team, there might be some issue with recent maintenance you're doing under the covers. We didn't change any code, the load pattern didn't change, and additionally we're seeing a several bugs: "An error occurred parsing (locally or remotely) the arguments to DATASTORE_V3.RunQuery(). "
I know that hiccups can happen on GAE, so it's not like I'm flagging this instantly. But the change doesn't go away, and 36h of decreased performance is starting to hurt. Here's one of the error stacktraces I'm seeing: Caused by: com.google.apphosting.api.ApiProxy$ArgumentException: An error occurred parsing (locally or remotely) the arguments to DATASTORE_V3.RunQuery(). at com.google.net.rpc3.client.RpcStub$RpcCallbackDispatcher$1.runInContext(RpcStub.java:781) at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:459) at com.google.tracing.TraceContext.runInContext(TraceContext.java:701) at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:336) at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:328) at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:456) at com.google.net.rpc3.client.RpcStub$RpcCallbackDispatcher.rpcFinished(RpcStub.java:823) at com.google.net.rpc3.client.RpcStub$RpcCallbackDispatcher.success(RpcStub.java:808) at com.google.net.rpc3.impl.client.RpcClientInternalContext.runCallbacks(RpcClientInternalContext.java:893) at com.google.net.rpc3.impl.client.RpcClientInternalContext.finishRpcAndNotifyApp(RpcClientInternalContext.java:798) at com.google.net.rpc3.impl.client.RpcNetChannel.afterFinishingActiveRpc(RpcNetChannel.java:1063) at com.google.net.rpc3.impl.client.RpcNetChannel.finishRpc(RpcNetChannel.java:911) at com.google.net.rpc3.impl.client.RpcNetChannel.handleResponse(RpcNetChannel.java:2265) at com.google.net.rpc3.impl.client.RpcNetChannel.messageReceived(RpcNetChannel.java:2068) at com.google.net.rpc3.impl.client.RpcNetChannel.access$2000(RpcNetChannel.java:143) at com.google.net.rpc3.impl.client.RpcNetChannel$TransportCallback.receivedMessage(RpcNetChannel.java:3127) at com.google.net.rpc3.impl.client.RpcChannelTransportData$TransportCallback.receivedMessage(RpcChannelTransportData.java:599) at com.google.net.rpc3.impl.wire.RpcBaseTransport.receivedMessage(RpcBaseTransport.java:417) at com.google.net.eventmanager.AbstractFutureTask$Sync.innerRun(AbstractFutureTask.java:260) at com.google.net.eventmanager.AbstractFutureTask.run(AbstractFutureTask.java:121) at com.google.net.eventmanager.EventManagerImpl.runTask(EventManagerImpl.java:578) at com.google.net.eventmanager.EventManagerImpl.internalRunWorkerLoop(EventManagerImpl.java:1002) at com.google.net.eventmanager.EventManagerImpl.runWorkerLoop(EventManagerImpl.java:884) at com.google.net.eventmanager.WorkerThreadInfo.runWorkerLoop(WorkerThreadInfo.java:136) at com.google.net.eventmanager.EventManagerImpl$WorkerThread.run(EventManagerImpl.java:1855) And here's a picture of the response time increase, taken about 24 hours ago: <https://lh4.googleusercontent.com/--Lf5oZY3xCc/UGyR0weY_SI/AAAAAAAAAEE/AZeG_6QToNM/s1600/Dashboard+-+Small+Improvements.jpg> Application ID is small-improvements-hrd, we're on HRD, and we were on F2 when this happened. Changing to F4 didn't make a difference, neither did restarting the instances. Which would have surprised me anyway, since it seems to be database related. I'd investigate further, but I'm on vacation. I hope this can be "solved" by moving us to a different cluster for now? Kind regards, Per -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/cyeWlHvzoDYJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
