Hi, We are using Jboss 4.0.5 in a cluster. We are experiencing deadlock within the Jboss Cache, which is causing out of memory problems on the server. The Jboss Cache Version is 1.4.1SP3 (bundled Cache for jboss 4.0.5). We only expirience this issue when BUDDY replication is enabled.
The thread dump reveils a lot of threads waiting on a lock for jgroups.. "http-0.0.0.0-9006-11" daemon prio=1 tid=0x086eb710 nid=0x719 waiting for monitor entry [0x87d44000..0x87d45600] at org.jgroups.protocols.UNICAST.down(UNICAST.java:265) - waiting to lock <0x9246a478> (a org.jgroups.protocols.UNICAST$Entry) at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517) at org.jgroups.stack.Protocol.passDown(Protocol.java:551) at org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:283) at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517) at org.jgroups.stack.Protocol.passDown(Protocol.java:551) at org.jgroups.protocols.pbcast.GMS.down(GMS.java:840) at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517) at org.jgroups.stack.Protocol.passDown(Protocol.java:551) at org.jgroups.protocols.FC.sendCreditRequest(FC.java:567) at org.jgroups.protocols.FC.handleDownMessage(FC.java:390) at org.jgroups.protocols.FC.down(FC.java:320) at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517) at org.jgroups.protocols.FC.receiveDownEvent(FC.java:314) at org.jgroups.stack.Protocol.passDown(Protocol.java:551) at org.jgroups.protocols.FRAG2.down(FRAG2.java:167) at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517) at org.jgroups.stack.Protocol.passDown(Protocol.java:551) at org.jgroups.protocols.pbcast.STATE_TRANSFER.down(STATE_TRANSFER.java:294) at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517) at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:385) at org.jgroups.JChannel.down(JChannel.java:1231) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:788) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.passDown(MessageDispatcher.java:765) at org.jgroups.blocks.RequestCorrelator.sendRequest(RequestCorrelator.java:295) at org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:444) at org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:193) at org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:429) at org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:192) at sun.reflect.GeneratedMethodAccessor177.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.cache.TreeCache.callRemoteMethodsViaReflection(TreeCache.java:4435) at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4391) at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4344) at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4455) at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:110) at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:88) at org.jboss.cache.interceptors.ReplicationInterceptor.handleReplicatedMethod(ReplicationInterceptor.java:119) at org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:88) at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68) at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365) at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160) at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68) at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:167) at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863) at org.jboss.cache.TreeCache.put(TreeCache.java:3759) at sun.reflect.GeneratedMethodAccessor186.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94) at org.jboss.mx.server.Invocation.invoke(Invocation.java:86) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659) at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210) at $Proxy57.put(Unknown Source) at org.jboss.web.tomcat.tc5.session.JBossCacheWrapper.put(JBossCacheWrapper.java:141) at org.jboss.web.tomcat.tc5.session.JBossCacheService.putSession(JBossCacheService.java:315) at org.jboss.web.tomcat.tc5.session.JBossCacheClusteredSession.processSessionRepl(JBossCacheClusteredSession.java:121) - locked <0x93ffa170> (a org.jboss.web.tomcat.tc5.session.SessionBasedClusteredSession) at org.jboss.web.tomcat.tc5.session.JBossCacheManager.processSessionRepl(JBossCacheManager.java:1093) at org.jboss.web.tomcat.tc5.session.JBossCacheManager.storeSession(JBossCacheManager.java:648) - locked <0x93ffa170> (a org.jboss.web.tomcat.tc5.session.SessionBasedClusteredSession) at org.jboss.web.tomcat.tc5.session.InstantSnapshotManager.snapshot(InstantSnapshotManager.java:49) at org.jboss.web.tomcat.tc5.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:98) at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105) at org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156) at org.apache.catalina.valves.FastCommonAccessLogValve.invoke(FastCommonAccessLogValve.java:495) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869) at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664) at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527) Here is the object that has the lock: http-0.0.0.0-9006-12" daemon prio=1 tid=0x8a563918 nid=0x71a waiting for monitor entry [0x87dc5000..0x87dc6480] at org.jgroups.stack.Retransmitter$Entry.(Retransmitter.java:272) at org.jgroups.stack.Retransmitter.add(Retransmitter.java:104) - locked <0x9246a670> (a java.util.LinkedList) at org.jgroups.stack.AckSenderWindow.add(AckSenderWindow.java:100) at org.jgroups.protocols.UNICAST.down(UNICAST.java:279) - locked <0x9246a478> (a org.jgroups.protocols.UNICAST$Entry) at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517) at org.jgroups.stack.Protocol.passDown(Protocol.java:551) at org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:283) at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517) at org.jgroups.stack.Protocol.passDown(Protocol.java:551) at org.jgroups.protocols.pbcast.GMS.down(GMS.java:840) at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517) at org.jgroups.stack.Protocol.passDown(Protocol.java:551) at org.jgroups.protocols.FC.sendCreditRequest(FC.java:567) at org.jgroups.protocols.FC.handleDownMessage(FC.java:390) at org.jgroups.protocols.FC.down(FC.java:320) at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517) at org.jgroups.protocols.FC.receiveDownEvent(FC.java:314) at org.jgroups.stack.Protocol.passDown(Protocol.java:551) at org.jgroups.protocols.FRAG2.down(FRAG2.java:167) at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517) at org.jgroups.stack.Protocol.passDown(Protocol.java:551) at org.jgroups.protocols.pbcast.STATE_TRANSFER.down(STATE_TRANSFER.java:294) at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517) at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:385) at org.jgroups.JChannel.down(JChannel.java:1231) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:788) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.passDown(MessageDispatcher.java:765) at org.jgroups.blocks.RequestCorrelator.sendRequest(RequestCorrelator.java:295) at org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:444) at org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:193) at org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:429) at org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:192) at sun.reflect.GeneratedMethodAccessor177.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.cache.TreeCache.callRemoteMethodsViaReflection(TreeCache.java:4435) at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4391) at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4344) at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4455) at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:110) at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:88) at org.jboss.cache.interceptors.ReplicationInterceptor.handleReplicatedMethod(ReplicationInterceptor.java:119) at org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:88) at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68) at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365) at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160) at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68) at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:167) at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863) at org.jboss.cache.TreeCache.put(TreeCache.java:3759) ... There are some JIRA's for Jboss Cache that appear to have a similar stack trace (533, 1103), but they are Caceloader related ? Any help ?!?! Thanks, David View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4069357#4069357 Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4069357 _______________________________________________ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user