Hi, I've been seeing the following phenomenon: Datastore or Memcache operations (e.g. PersistentManager#getObjectById, Query#execute, or Cache#get/put) freezes repeatedly for a few certain users (source IP address), while all the other requests are not affected. Sometimes it lasts just for a 10 - 15 minutes, and sometimes it lasts a few hours. As a result, those phenomenons becomes a major part of the CPU cost of our app.
As the failures are repeating, I don't think it is the 0.1% spontaneous failures mentioned in the past discussions. I suspect that a certain App Server node or cluster would be having a trouble on its tablet servers or memcache service, and it is taking so long time to fail over. What should I do for this? Thanks, Kaz Example logs: ----- com.google.apphosting.runtime.HardDeadlineExceededError: This request (f2e1077c0b151110) started at 2009/09/24 13:14:29.281 UTC and was still executing at 2009/09/24 13:14:59.071 UTC. at com.google.appengine.runtime.Request.process-c9366ab62e22ae39 (Request.java) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos (Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos (Unknown Source) at java.util.concurrent.CountDownLatch.await(Unknown Source) at com.google.net.rpc.util.RpcWaiter.waitForRpcsToFinish (RpcWaiter.java:96) at com.google.apphosting.api.ApiProxy.makeSyncCall(ApiProxy.java:79) at com.google.appengine.api.datastore.DatastoreApiHelper.makeSyncCall (DatastoreApiHelper.java:48) at com.google.appengine.api.datastore.DatastoreServiceImpl$1.run (DatastoreServiceImpl.java:110) at com.google.appengine.api.datastore.TransactionRunner.runInTransaction (TransactionRunner.java:30) at com.google.appengine.api.datastore.DatastoreServiceImpl.get (DatastoreServiceImpl.java:96) at com.google.appengine.api.datastore.DatastoreServiceImpl.get (DatastoreServiceImpl.java:69) at com.google.appengine.api.datastore.DatastoreServiceImpl.get (DatastoreServiceImpl.java:62) at org.datanucleus.store.appengine.RuntimeExceptionWrappingDatastoreService.get (RuntimeExceptionWrappingDatastoreService.java:53) at org.datanucleus.store.appengine.DatastorePersistenceHandler.get (DatastorePersistenceHandler.java:79) at org.datanucleus.store.appengine.DatastorePersistenceHandler.get (DatastorePersistenceHandler.java:91) at org.datanucleus.store.appengine.DatastorePersistenceHandler.locateObject (DatastorePersistenceHandler.java:559) at org.datanucleus.state.JDOStateManagerImpl.validate (JDOStateManagerImpl.java:4273) at org.datanucleus.ObjectManagerImpl.findObject (ObjectManagerImpl.java:2444) at org.datanucleus.jdo.JDOPersistenceManager.getObjectById (JDOPersistenceManager.java:1671) at org.datanucleus.jdo.JDOPersistenceManager.getObjectById (JDOPersistenceManager.java:1767) at org.datanucleus.store.appengine.jdo.DatastoreJDOPersistenceManager.getObjectById (DatastoreJDOPersistenceManager.java:71) ----- com.google.apphosting.runtime.HardDeadlineExceededError: This request (517b68e8231edfc1) started at 2009/09/24 13:12:52.711 UTC and was still executing at 2009/09/24 13:13:28.846 UTC. at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos (Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos (Unknown Source) at java.util.concurrent.CountDownLatch.await(Unknown Source) at com.google.net.rpc.util.RpcWaiter.waitForRpcsToFinish (RpcWaiter.java:96) at com.google.apphosting.runtime.ApiProxyImpl.doSyncCall (ApiProxyImpl.java:99) at com.google.apphosting.runtime.ApiProxyImpl.access$000 (ApiProxyImpl.java:36) at com.google.apphosting.runtime.ApiProxyImpl$1.run(ApiProxyImpl.java: 68) at com.google.apphosting.runtime.ApiProxyImpl$1.run(ApiProxyImpl.java: 64) at java.security.AccessController.doPrivileged(Native Method) at com.google.apphosting.runtime.ApiProxyImpl.makeSyncCall (ApiProxyImpl.java:64) at com.google.apphosting.runtime.ApiProxyImpl.makeSyncCall (ApiProxyImpl.java:36) at com.google.apphosting.api.ApiProxy.makeSyncCall(ApiProxy.java:79) at com.google.appengine.api.memcache.MemcacheServiceImpl.makeSyncCall (MemcacheServiceImpl.java:162) at com.google.appengine.api.memcache.MemcacheServiceImpl.get (MemcacheServiceImpl.java:233) at com.google.appengine.api.memcache.stdimpl.GCache.get(GCache.java: 157) --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~----------~----~----~----~------~----~------~--~---