Hello all
I´m Mikel, and me and my workmates have already been a while testing our
environment in order to establish a in memory session replication cluster
for our servers. The thing is that our servers are often loaded with up to a
thousand (1000) and more sessions (and now, we have three tomcat nodes in
different machines working in a load balancer!!!!). In hot moments, I have
counted up to 4500 sessions distributed between the three servers.
So, I'll tell you the main configuration our production environment. Web
servers:
Two windows server 2003 with IIS and isapi_redirect.dll conector.
App servers
node 1 & 2: IBM Xseries_3550 Intel Xeon CPU 5150 @2,66GHz, 2,00 GB RAM,
Windows 2003 Server R2
node3: IBM XSeries_366 Intel Xeon CPU 3,20Ghz, 3,00 GB RAM, Windows Server
2003 R2
In our development environment, where we have been making our tests, we have
2 tomcats in two different machines
node 1: Intel Xeon CPU E5440 @ 2,83GHz, 1,00 GB RAM, Windows Server 2003 R2
node 2: IBM XSeries_3550 Intel Xeon CPU E5440 @ 2,83 GHz, 2GB RAM, Windows
Server 2003 R2
We have been testing the Tomcat 5.5.9 that we are using in production,
finding some trouble, even after aplying the clustering fix pack from
https://issues.apache.org/bugzilla/show_bug.cgi?id=34389 . Finally, we took
the decission to upgrade to last Tomcat 6 available, version 6.0.18, to see
if the announced refactoring of cluster subsystem could solve our trouble.
After all this prety long intro, I'll tell you the reason of my request.
When we test the cluster (in development environment) with both nodes
running, we create up to a thousand sessions, keeping alive and modificating
about 500. So, we can see that all of them get replicated to the other node
quickly . The trouble comes when, after shutting down one of the instances,
we start it again (while the half of the alive sessions are still beeing
modified by the JMeter test). In our tests, the starting tomcat instance
finally gets hunged when receiving sessions from the alive node.
Theese are the traces seen in the catalina.log of node 1 when starting it:
Jan 26, 2009 6:51:56 PM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal
performance in production environments was not found on the
java.library.path:
C:\tomcat-6.0.18\bin;.;C:\WINDOWS\system32;C:\WINDOWS;C:\Program
Files\Serena\Dimensions
10.1\CM\prog;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\system32\WBEM;C:\Program
Files\IBM\Director\bin;C:\Program Files\Common
Files\IBM\ICC\cimom\bin;C:\Program Files\System Center Operations Manager
2007\
Jan 26, 2009 6:51:56 PM org.apache.coyote.http11.Http11Protocol init
INFO: Initializing Coyote HTTP/1.1 on http-9080
Jan 26, 2009 6:51:56 PM org.apache.coyote.http11.Http11Protocol init
INFO: Initializing Coyote HTTP/1.1 on http-9081
Jan 26, 2009 6:51:56 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 1928 ms
Jan 26, 2009 6:51:56 PM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
Jan 26, 2009 6:51:56 PM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.18
Jan 26, 2009 6:51:56 PM org.apache.catalina.ha.tcp.SimpleTcpCluster start
INFO: Cluster is about to start
Jan 26, 2009 6:51:56 PM org.apache.catalina.tribes.transport.ReceiverBase
bind
INFO: Receiver Server Socket bound to:/172.26.102.233:4009
Jan 26, 2009 6:51:56 PM
org.apache.catalina.tribes.membership.McastServiceImpl setupSocket
INFO: Attempting to bind the multicast socket to /228.0.0.9:45569
Jan 26, 2009 6:51:56 PM
org.apache.catalina.tribes.membership.McastServiceImpl setupSocket
INFO: Binding to multicast address, failed. Binding to port only.
Jan 26, 2009 6:51:56 PM
org.apache.catalina.tribes.membership.McastServiceImpl setupSocket
INFO: Setting multihome multicast interface to:/172.26.102.233
Jan 26, 2009 6:51:56 PM
org.apache.catalina.tribes.membership.McastServiceImpl setupSocket
INFO: Setting cluster mcast soTimeout to 1000
Jan 26, 2009 6:51:56 PM
org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
INFO: Sleeping for 2000 milliseconds to establish cluster membership, start
level:4
Jan 26, 2009 6:51:57 PM org.apache.catalina.ha.tcp.SimpleTcpCluster
memberAdded
INFO: Replication member
added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{-84, 26, 102,
-60}:4009,{-84, 26, 102, -60},4009, alive=1938953,id={-57 67 34 -23 -38 83
74 68 -67 -87 -112 -94 13 102 -78 -20 }, payload={}, command={}, domain={},
]
Jan 26, 2009 6:51:58 PM
org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
INFO: Done sleeping, membership established, start level:4
Jan 26, 2009 6:51:58 PM
org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
INFO: Sleeping for 2000 milliseconds to establish cluster membership, start
level:8
Jan 26, 2009 6:51:58 PM org.apache.catalina.tribes.io.BufferPool
getBufferPool
INFO: Created a buffer pool with max size:104857600 bytes of
type:org.apache.catalina.tribes.io.BufferPool15Impl
Jan 26, 2009 6:52:00 PM
org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
INFO: Done sleeping, membership established, start level:8
Jan 26, 2009 6:52:14 PM org.apache.catalina.ha.session.DeltaManager start
INFO: Register manager to cluster element Engine with name Catalina
Jan 26, 2009 6:52:14 PM org.apache.catalina.ha.session.DeltaManager start
INFO: Starting clustering manager at
Jan 26, 2009 6:52:14 PM org.apache.catalina.ha.session.DeltaManager
getAllClusterSessions
WARNING: Manager [localhost#], requesting session state from
org.apache.catalina.tribes.membership.MemberImpl[tcp://{-84, 26, 102,
-60}:4009,{-84, 26, 102, -60},4009, alive=1955953,id={-57 67 34 -23 -38 83
74 68 -67 -87 -112 -94 13 102 -78 -20 }, payload={}, command={}, domain={},
]. This operation will timeout if no session state has been received within
-1 seconds.
Jan 26, 2009 7:00:50 PM org.apache.catalina.startup.Catalina stopServer
SEVERE: Catalina.stop:
java.net.ConnectException: Connection refused: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:364)
at java.net.Socket.connect(Socket.java:507)
at java.net.Socket.connect(Socket.java:457)
at java.net.Socket.<init>(Socket.java:365)
at java.net.Socket.<init>(Socket.java:178)
at org.apache.catalina.startup.Catalina.stopServer(Catalina.java:421)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.catalina.startup.Bootstrap.stopServer(Bootstrap.java:337)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:415)
The last trace was because after nearly ten minutes waiting, we resolved to
shut the tomcat instance down. In the running Tomcat (node 2):
26-ene-2009 18:51:58 org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
INFO: Replication member
added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{-84, 26, 102,
-23}:4009,{-84, 26, 102, -23},4009, alive=2047,id={-20 60 112 -113 35 -41 71
-5 -124 47 93 -37 117 -9 -9 29 }, payload={}, command={}, domain={}, ]
26-ene-2009 19:00:51
org.apache.catalina.tribes.transport.nio.NioReplicationTask run
ADVERTENCIA: IOException in replication worker, unable to drain channel.
Probable cause: Keep alive socket closed[An existing connection was forcibly
closed by the remote host].
26-ene-2009 19:00:53
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Received
memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{-84,
26, 102, -23}:4009,{-84, 26, 102, -23},4009, alive=533282,id={-20 60 112
-113 35 -41 71 -5 -124 47 93 -37 117 -9 -9 29 }, payload={}, command={},
domain={}, ]] message. Will verify.
26-ene-2009 19:00:54
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Verification complete. Member
disappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{-84, 26,
102, -23}:4009,{-84, 26, 102, -23},4009, alive=533282,id={-20 60 112 -113 35
-41 71 -5 -124 47 93 -37 117 -9 -9 29 }, payload={}, command={}, domain={},
]]
26-ene-2009 19:00:54 org.apache.catalina.ha.tcp.SimpleTcpCluster
memberDisappeared
INFO: Received member
disappeared:org.apache.catalina.tribes.membership.MemberImpl[tcp://{-84, 26,
102, -23}:4009,{-84, 26, 102, -23},4009, alive=533282,id={-20 60 112 -113 35
-41 71 -5 -124 47 93 -37 117 -9 -9 29 }, payload={}, command={}, domain={},
]
Our Cluster config in the nodes (as you can see, we configured the cluster
at engine level) :
Node 1:
<Engine name="Catalina" defaultHost="localhost" jvmRoute="worker62">
<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster">
<Manager className="org.apache.catalina.ha.session.DeltaManager"
name="clusterPruebas6"
stateTransferTimeout="-1"
expireSessionsOnShutdown="false"
notifyListenersOnReplication="true"/>
<Channel className="org.apache.catalina.tribes.group.GroupChannel">
<Membership
className="org.apache.catalina.tribes.membership.McastService"
address="228.0.0.9"
bind="172.26.102.233"
port="45569"
frequency="1000"
dropTime="3000"/>
<Receiver
className="org.apache.catalina.tribes.transport.nio.NioReceiver"
address="172.26.102.233"
port="4009"
autoBind="100"
selectorTimeout="5000"
maxThreads="12"/>
<Sender
className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
<Transport
className="org.apache.catalina.tribes.transport.nio.PooledParallelSender" />
</Sender>
<Interceptor
className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
<Interceptor
className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
</Channel>
</Cluster>
...
Node 2:
<Engine name="Catalina" defaultHost="localhost" jvmRoute="worker66">
<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster">
<Manager
className="org.apache.catalina.ha.session.DeltaManager"
name="clusterPruebas6"
stateTransferTimeout="-1"
expireSessionsOnShutdown="false"
notifyListenersOnReplication="true"/>
<Channel
className="org.apache.catalina.tribes.group.GroupChannel">
<Membership
className="org.apache.catalina.tribes.membership.McastService"
address="228.0.0.9"
bind="172.26.102.196"
port="45569"
frequency="1000"
dropTime="5000"/>
<Receiver
className="org.apache.catalina.tribes.transport.nio.NioReceiver"
address="172.26.102.196"
port="4009"
autoBind="100"
selectorTimeout="100"
maxThreads="12"/>
<Sender
className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
<Transport
className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
maxRetryAttempts="0" />
</Sender>
<Interceptor
className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
<Interceptor
className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
</channel>
</Cluster>
So, after watching all this info, here go my questions.
1) Considering the high load of our servers, and even when we think that the
in memory replication matches our expectatives more than database or in file
persistence, is it possible to join a in memory replication or is it
discouraged?
2) In our cluster configuration we have been testing with the
stateTransferTimeout set to -1 to avoid quitting the DeltaManager
getAllClusterSessions, because it is very important for us to replicate all
of the sessions to the starting node. Anyway, should we set any other value
here?
3) Any other sugestion to our configuration?
Thank you very much.
Mikel Ibiricu
