I am experiencing what appears to be a deadlock situation in JBoss Cache 1.4.1.SP3. After some period of time with the app up and running in JBoss AS 4.0.5 I am seeing one thread holding a couple of monitor locks in BoundedLinkedQueue and a multitude of threads waiting on the lock. The application grinds to a halt with users experiencing hanging HTTP responses.
In the latest scenario I have 28 HTTP threads, 26 are blocked (see Stack #1). 1 thread holds 2 locks (see Stack #2). 1 thread is still free (see Stack #3). A Google search indicates that BoundedLinkedQueue has had issues with race conditions in the past, http://altair.cs.oswego.edu/pipermail/concurrency-interest/2004-September/001037.html, although the scenario seems slightly different. Stack #1 "http-0.0.0.0-8080-12" daemon prio=1 tid=0x08c5bd58 nid=0x26e0 waiting for monitor entry [0x97cfc000..0x97cff030] | at EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue.put(Unknown Source) | - waiting to lock <0xb4291690> (a java.lang.Object) | at org.jboss.cache.eviction.Region.putNodeEvent(Region.java:141) | at org.jboss.cache.interceptors.EvictionInterceptor.doEventUpdatesOnRegionManager(EvictionInterceptor.java:149) | at org.jboss.cache.interceptors.EvictionInterceptor.updateNode(EvictionInterceptor.java:122) | at org.jboss.cache.interceptors.EvictionInterceptor.invoke(EvictionInterceptor.java:97) | at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68) | at org.jboss.cache.interceptors.OptimisticCreateIfNotExistsInterceptor.invoke(OptimisticCreateIfNotExistsInterceptor.java:69) | at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68) | at org.jboss.cache.interceptors.OptimisticValidatorInterceptor.invoke(OptimisticValidatorInterceptor.java:87) | at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68) | at org.jboss.cache.interceptors.OptimisticLockingInterceptor.invoke(OptimisticLockingInterceptor.java:126) | at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68) | at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365) | at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160) | at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68) | at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:138) | at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863) | at org.jboss.cache.TreeCache.get(TreeCache.java:3627) | at org.jboss.cache.TreeCache.get(TreeCache.java:3608) | Stack #2 "http-0.0.0.0-8080-11" daemon prio=1 tid=0x087dab18 nid=0x2691 in Object.wait() [0x976f0000..0x976f3030] | at java.lang.Object.wait(Native Method) | at java.lang.Object.wait(Object.java:474) | at EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue.put(Unknown Source) | - locked <0xb42914e8> (a EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue) | - locked <0xb4291690> (a java.lang.Object) | at org.jboss.cache.eviction.Region.putNodeEvent(Region.java:141) | at org.jboss.cache.interceptors.EvictionInterceptor.doEventUpdatesOnRegionManager(EvictionInterceptor.java:149) | at org.jboss.cache.interceptors.EvictionInterceptor.updateNode(EvictionInterceptor.java:122) | at org.jboss.cache.interceptors.EvictionInterceptor.invoke(EvictionInterceptor.java:97) | at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68) | at org.jboss.cache.interceptors.OptimisticCreateIfNotExistsInterceptor.invoke(OptimisticCreateIfNotExistsInterceptor.java:69) | at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68) | at org.jboss.cache.interceptors.OptimisticValidatorInterceptor.invoke(OptimisticValidatorInterceptor.java:87) | at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68) | at org.jboss.cache.interceptors.OptimisticLockingInterceptor.invoke(OptimisticLockingInterceptor.java:126) | at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68) | at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365) | at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160) | at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68) | at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:138) | at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863) | at org.jboss.cache.TreeCache.get(TreeCache.java:3627) | at org.jboss.cache.TreeCache.get(TreeCache.java:3608) | Stack #3 "http-0.0.0.0-8080-27" daemon prio=1 tid=0x09643720 nid=0x2fc0 in Object.wait() [0x978f7000..0x978f71b0] | at java.lang.Object.wait(Native Method) | - waiting on <0xc3c65dc0> (a org.apache.tomcat.util.net.MasterSlaveWorkerThread) | at java.lang.Object.wait(Object.java:474) | at org.apache.tomcat.util.net.MasterSlaveWorkerThread.await(MasterSlaveWorkerThread.java:81) | - locked <0xc3c65dc0> (a org.apache.tomcat.util.net.MasterSlaveWorkerThread) | at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:107) | at java.lang.Thread.run(Thread.java:595) | TreeCache Configuration <server> | <classpath codebase="./lib" archives="jboss-cache.jar, jgroups.jar"/> | <mbean code="org.jboss.cache.TreeCache" name="jboss.cache:service=TreeCache"> | <depends>jboss:service=Naming</depends> | <depends>jboss:service=TransactionManager</depends> | <attribute name="ClusterName">FCXp_hibernate</attribute> | <attribute name="CacheMode">LOCAL</attribute> | <attribute name="SyncReplTimeout">10000</attribute> | <attribute name="LockAcquisitionTimeout">15000</attribute> | <attribute name="FetchInMemoryState">false</attribute> | <attribute name="NodeLockingScheme">OPTIMISTIC</attribute> | | <attribute name="EvictionPolicyClass"> org.jboss.cache.eviction.LRUPolicy </attribute> | <attribute name="EvictionPolicyConfig"> | <config> | <attribute name="wakeUpIntervalSeconds">10000</attribute> | <!-- Cache wide default --> | <region name="/_default_"> | <attribute name="maxNodes">200000</attribute> | <attribute name="timeToIdleSeconds">10000</attribute> | </region> | </config> | </attribute> | </mbean> | </server> View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4065227#4065227 Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4065227 _______________________________________________ jboss-user mailing list [email protected] https://lists.jboss.org/mailman/listinfo/jboss-user
