https://issues.apache.org/bugzilla/show_bug.cgi?id=52365
Bug #: 52365 Summary: tomcat cluster org.apache.catalina.tribes.ChannelException: Sender not connected Product: Tomcat 6 Version: 6.0.32 Platform: PC OS/Version: Linux Status: NEW Severity: regression Priority: P2 Component: Cluster AssignedTo: dev@tomcat.apache.org ReportedBy: dhor...@gmail.com Classification: Unclassified We have got tomcat cluster of two nodes (each on its own server ), configured to use DeltaManger. Our Snder configure as following on both tomcats: <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter"> <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender" timeout="6000" maxRetryAttempts="2" soKeepAlive="true" soLingerTime="6" soTrafficClass="0x08" /> </Sender> When we stop one of the tomcats and the other one is still trying to sent session delta information and fails with the following error: 18-Dec-2011 21:48:05.648 FINE [pool-2-thread-4] org.apache.catalina.tribes.transport.nio.NioReplicationTask.remoteEof Channel closed on the remote end, disconnecting 18-Dec-2011 21:48:05.650 FINE [pool-2-thread-5] org.apache.catalina.tribes.transport.nio.NioReplicationTask.remoteEof Channel closed on the remote end, disconnecting 18-Dec-2011 21:48:06.114 FINE [http-8080-1] org.apache.catalina.ha.session.DeltaManager.sendCreateSession Manager [/myaccount] send new session (CE176CCE1BF31C41977B6D43891EFB36.) 18-Dec-2011 21:48:06.115 FINE [http-8080-1] org.apache.catalina.ha.session.DeltaManager.createSession Created a DeltaSession with Id [CE176CCE1BF31C41977B6D43891EFB36.] Total count=276 18-Dec-2011 21:48:06.117 WARNING [pool-1-thread-1] org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop Member send is failing for:tcp://{10, 210, 160, 71}:4205 ; Setting to suspect and retrying. 18-Dec-2011 21:48:06.121 FINE [pool-1-thread-1] org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendAsyncData Error while processing async message. org.apache.catalina.tribes.ChannelException: Send failed, attempt:3 max:2; Faulty members:tcp://{10, 210, 160, 71}:4205; at org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:172) at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:78) at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:53) at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:81) at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78) at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) at org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor.sendMessage(ThroughputInterceptor.java:61) at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendAsyncData(MessageDispatchInterceptor.java:178) at org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor$1.run(MessageDispatch15Interceptor.java:64) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) at org.apache.catalina.tribes.transport.nio.NioSender.process(NioSender.java:88) at org.apache.catalina.tribes.transport.nio.ParallelNioSender.doLoop(ParallelNioSender.java:130) ... 12 more And then ,when we start the other tomcat , Sender never gets connected again. 18-Dec-2011 21:48:06.297 FINE [http-8080-1] org.apache.catalina.ha.session.DeltaManager.sendCreateSession Manager [/myaccount] send new session (A6955A82C56920EABFFA5096BA4ECD8C.) 18-Dec-2011 21:48:06.297 FINE [http-8080-1] org.apache.catalina.ha.session.DeltaManager.createSession Created a DeltaSession with Id [A6955A82C56920EABFFA5096BA4ECD8C.] Total count=277 18-Dec-2011 21:48:06.298 FINE [pool-1-thread-2] org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendAsyncData Error while processing async message. org.apache.catalina.tribes.ChannelException: Sender not connected.; No faulty members identified. at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:45) at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:81) at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78) at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) at org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor.sendMessage(ThroughputInterceptor.java:61) at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75) at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendAsyncData(MessageDispatchInterceptor.java:178) at org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor$1.run(MessageDispatch15Interceptor.java:64) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) So whenever we restart one of the tomcats the other keeps getting 'Sender not connected' exceptions and to re-connect the Sender we need to restart that tomcat instance. But when we do that the other tomcat instance gets the same problem. So cluster never works The problem seems to happen in PooledParallelSender.sendMessage(...) public void sendMessage(Member[] destination, ChannelMessage message) throws ChannelException { if ( !connected ) throw new ChannelException("Sender not connected."); ... That if statement makes sure Sender never gets connected again, unless we restart Please advise -- Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug. --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org