Production cluster crashed with "SEVERE: TCP Worker thread..." error

[Joseph Lam]

Hi

In my 4-node TC5.5.9 cluster, the catalina.out suddenly got flooded by the
error below and the whole cluster simply stopped responding to web
clients. I have to restart all nodes in order to get it back to normal. It
seemed to be triggerred by a surge in loading but not able to resume even
after the load was reduced later on. The cluster has been working fine
previously.

catalina.out:

SEVERE: TCP Worker thread in cluster caught 'java.io.IOException:
Connection reset by peer' closing channel
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcher.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
        at sun.nio.ch.IOUtil.read(IOUtil.java:206)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:207)
        at
org.apache.catalina.cluster.tcp.TcpReplicationThread.drainChannel(TcpReplicationThread.java:123)
        at
org.apache.catalina.cluster.tcp.TcpReplicationThread.run(TcpReplicationThread.java:67)
...

Here is my server.xml <Cluster> config:
        <Cluster className="org.apache.catalina.cluster.tcp.SimpleTcpCluster"
                 managerClassName="qpi.bochk.web.ClusterSessionManager"
                 expireSessionsOnShutdown="false"
                 useDirtyFlag="true"
                 notifyListenersOnReplication="true">

            <Membership
                className="org.apache.catalina.cluster.mcast.McastService"
                mcastAddr="228.0.0.6"
                mcastPort="45564"
                mcastFrequency="500"
                mcastDropTime="3000"/>

            <Receiver
                className="org.apache.catalina.cluster.tcp.ReplicationListener"
                tcpListenAddress="auto"
                tcpListenPort="4001"
                tcpSelectorTimeout="100"
                tcpThreadCount="10"/>

            <Sender
                
className="org.apache.catalina.cluster.tcp.ReplicationTransmitter"
                replicationMode="pooled"
                ackTimeout="15000"/>

            <Valve
className="org.apache.catalina.cluster.tcp.ReplicationValve"
                   
filter=".*\.gif;.*\.js;.*\.jpg;.*\.htm;.*\.html;.*\.css;.*\.CSS;.*\.GIF;.*\.JPG;.*\.txt;"/>

            <Deployer
className="org.apache.catalina.cluster.deploy.FarmWarDeployer"
                      tempDir="/tmp/war-temp/"
                      deployDir="/tmp/war-deploy/"
                      watchDir="/tmp/war-listen/"
                      watchEnabled="false"/>
        </Cluster>

Regards,
Joseph Lam


---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]