[appengine-java] Re: Increased reliability issues on AppEngine for the past 48h
I am seeing inexplicable datastore timeouts, too. com.google.appengine.api.datastore.DatastoreTimeoutException: Unknown This happens for simple look-ups (checking if any account Kinds exist which correspond to the submitted cookie). It always goes away if I refresh the page. The error is very sporadic. On Jan 8, 12:54 pm, Jerome Mouton jerome.mou...@gmail.com wrote: Hello, Our AppEngine application has been experiencing an increase rate of bogus exceptions for the past 48+ hours. See the attached image for our requests/seconds and errors/seconds for the last 6 hours. We have been seeing the exact same pattern for the past 2 days. This is an app with billing enabled, and we have way (over 10x) below our billing quotas. The only change we can figure out that started at about the same time as this increase rate of error was the usage of the XMPP API being more frequent than before, but the exceptions are not related to the XMPP API calls, they are the one of the following: - com.google.appengine.api.datastore.DatastoreTimeoutException: Unknown on servlets data store JDO queries that are expected to be (and most of the time are) extremely fast; when it fails, the servlet shows times in this area: 4135ms 187cpu_ms 12api_cpu_ms - com.google.apphosting.api.ApiProxy$ApiDeadlineExceededException: The API call datastore_v3.RunQuery() took too long to respond and was cancelled. on servlets data store JDO queries that are expected to be (and most of the time are) extremely fast; when it fails, the servlet shows times in this area: 5464ms 194cpu_ms - Request was aborted after waiting too long to attempt to service your request. Most likely, this indicates that you have reached your simultaneous dynamic request limit. This is almost always due to excessively high latency in your app. Please seehttp://code.google.com/appengine/docs/quotas.htmlfor more details. Is this reliability degradation for the past 48 hours seen by other users? If a Googler needs our appId for review, please drop me an email. Thank you. Jerome 20100108-appengine_issue.png 80KViewDownload -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Increased reliability issues on AppEngine for the past 48h
I forgot to add that as you can see on the graph that is attached to the previous email, our request/sec rate which has been at about 5 queries/sec for the past few weeks is now showing spikes at 10 queries per sec, followed by drops to 0 queries per sec. There is no logical way to explain this, we should still have a pretty much constant 5 reqs/sec rate. The error rate spikes do not correlate to the reqs/sec unexpected spikes though. Let me know if I can provide additional data for review. Thanks Jerome On Jan 8, 6:54 pm, Jerome Mouton jerome.mou...@gmail.com wrote: Hello, Our AppEngine application has been experiencing an increase rate of bogus exceptions for the past 48+ hours. See the attached image for our requests/seconds and errors/seconds for the last 6 hours. We have been seeing the exact same pattern for the past 2 days. This is an app with billing enabled, and we have way (over 10x) below our billing quotas. The only change we can figure out that started at about the same time as this increase rate of error was the usage of the XMPP API being more frequent than before, but the exceptions are not related to the XMPP API calls, they are the one of the following: - com.google.appengine.api.datastore.DatastoreTimeoutException: Unknown on servlets data store JDO queries that are expected to be (and most of the time are) extremely fast; when it fails, the servlet shows times in this area: 4135ms 187cpu_ms 12api_cpu_ms - com.google.apphosting.api.ApiProxy$ApiDeadlineExceededException: The API call datastore_v3.RunQuery() took too long to respond and was cancelled. on servlets data store JDO queries that are expected to be (and most of the time are) extremely fast; when it fails, the servlet shows times in this area: 5464ms 194cpu_ms - Request was aborted after waiting too long to attempt to service your request. Most likely, this indicates that you have reached your simultaneous dynamic request limit. This is almost always due to excessively high latency in your app. Please seehttp://code.google.com/appengine/docs/quotas.htmlfor more details. Is this reliability degradation for the past 48 hours seen by other users? If a Googler needs our appId for review, please drop me an email. Thank you. Jerome 20100108-appengine_issue.png 80KViewDownload -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Increased reliability issues on AppEngine for the past 48h
What's your app id? (privately if you'd like) On Fri, Jan 8, 2010 at 11:58 AM, Jerome jerome.mou...@gmail.com wrote: I forgot to add that as you can see on the graph that is attached to the previous email, our request/sec rate which has been at about 5 queries/sec for the past few weeks is now showing spikes at 10 queries per sec, followed by drops to 0 queries per sec. There is no logical way to explain this, we should still have a pretty much constant 5 reqs/sec rate. The error rate spikes do not correlate to the reqs/sec unexpected spikes though. Let me know if I can provide additional data for review. Thanks Jerome On Jan 8, 6:54 pm, Jerome Mouton jerome.mou...@gmail.com wrote: Hello, Our AppEngine application has been experiencing an increase rate of bogus exceptions for the past 48+ hours. See the attached image for our requests/seconds and errors/seconds for the last 6 hours. We have been seeing the exact same pattern for the past 2 days. This is an app with billing enabled, and we have way (over 10x) below our billing quotas. The only change we can figure out that started at about the same time as this increase rate of error was the usage of the XMPP API being more frequent than before, but the exceptions are not related to the XMPP API calls, they are the one of the following: - com.google.appengine.api.datastore.DatastoreTimeoutException: Unknown on servlets data store JDO queries that are expected to be (and most of the time are) extremely fast; when it fails, the servlet shows times in this area: 4135ms 187cpu_ms 12api_cpu_ms - com.google.apphosting.api.ApiProxy$ApiDeadlineExceededException: The API call datastore_v3.RunQuery() took too long to respond and was cancelled. on servlets data store JDO queries that are expected to be (and most of the time are) extremely fast; when it fails, the servlet shows times in this area: 5464ms 194cpu_ms - Request was aborted after waiting too long to attempt to service your request. Most likely, this indicates that you have reached your simultaneous dynamic request limit. This is almost always due to excessively high latency in your app. Please seehttp:// code.google.com/appengine/docs/quotas.htmlfor more details. Is this reliability degradation for the past 48 hours seen by other users? If a Googler needs our appId for review, please drop me an email. Thank you. Jerome 20100108-appengine_issue.png 80KViewDownload -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
