Hi List,
Since a few weeks now, I started to notice in the catalina.out log file
messages regarding the cluster's operatability.
It reports that a member or members in my cluster have disappeared, and
appeared (member still alive) again. That's reasonable...the strange thing
is that they occur at exactly the same time.
Sometimes this message appears several times in a minute, and sometimes it
doesn't appear for several minutes.
I'm using Tomcat 6.0.18 with JDK 1.6.0_14 64bit on RedHat Linux 5.2.
I'm thinking about several posibilities here, as for example a bad switch or
something. Although, the thing is, I have other Tomcat instances in the same
network that do not show this behavior.
I would really apreciate it if someone could shed a light on what I can
investigate next.
As an example, I paste some output from my log:
Apr 29, 2010 1:43:31 PM org.apache.catalina.tribes.
group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received
memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
-68, 4, -36}:4003,{10, -68, 4, -36},4003, alive=23576482,id={17 96 -31 30
-112 76 73 -38 -87 6 -74 21 -124 117 18 -66 }, payload={}, command={},
domain={}, ]] message. Will verify.
Apr 29, 2010 1:43:31 PM
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Verification complete. Member still
alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, -68, 4,
-36}:4003,{10, -68, 4, -36},4003, alive=23576482,id={17 96 -31 30 -112 76 73
-38 -87 6 -74 21 -124 117 18 -66 }, payload={}, command={}, domain={}, ]]
Apr 29, 2010 1:47:59 PM
org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
WARNING: Member send is failing for:tcp://{10, -68, 4, -37}:4003 ; Setting
to suspect and retrying.
Apr 29, 2010 1:47:59 PM
org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
WARNING: Not retrying send for:tcp://{10, -68, 4, -37}:4003; Sender is
disconnected.
Apr 29, 2010 1:47:59 PM
org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
WARNING: Not retrying send for:tcp://{10, -68, 4, -37}:4003; Sender is
disconnected.
Apr 29, 2010 1:48:01 PM org.apache.catalina.ha.tcp.SimpleTcpCluster
memberDisappeared
INFO: Received member
disappeared:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, -68,
4, -37}:4003,{10, -68, 4, -37},4003, alive=23477210,id={96 77 36 -51 -85 -17
67 -53 -107 -22 9 77 -71 78 -106 -112 }, payload={}, command={}, domain={},
]
Apr 29, 2010 1:48:01 PM
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
performBasicCheck
INFO: Suspect member, confirmed
dead.[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, -68, 4,
-37}:4003,{10, -68, 4, -37},4003, alive=23477210,id={96 77 36 -51 -85 -17 67
-53 -107 -22 9 77 -71 78 -106 -112 }, payload={}, command={}, domain={}, ]]
Apr 29, 2010 1:48:08 PM
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Received
memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
-68, 4, -37}:4003,{10, -68, 4, -37},4003, alive=24454627,id={96 77 36 -51
-85 -17 67 -53 -107 -22 9 77 -71 78 -106 -112 }, payload={}, command={},
domain={}, ]] message. Will verify.
Apr 29, 2010 1:48:08 PM
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Verification complete. Member
disappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, -68,
4, -37}:4003,{10, -68, 4, -37},4003, alive=24454627,id={96 77 36 -51 -85 -17
67 -53 -107 -22 9 77 -71 78 -106 -112 }, payload={}, command={}, domain={},
]]
Apr 29, 2010 1:48:08 PM org.apache.catalina.ha.tcp.SimpleTcpCluster
memberDisappeared
INFO: Received member
disappeared:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, -68,
4, -37}:4003,{10, -68, 4, -37},4003, alive=24454627,id={96 77 36 -51 -85 -17
67 -53 -107 -22 9 77 -71 78 -106 -112 }, payload={}, command={}, domain={},
]
Apr 29, 2010 1:48:09 PM
org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
WARNING: Member send is failing for:tcp://{10, -68, 4, -37}:4003 ; Setting
to suspect and retrying.
Apr 29, 2010 1:48:09 PM
org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
WARNING: Not retrying send for:tcp://{10, -68, 4, -37}:4003; Sender is
disconnected.
Apr 29, 2010 1:48:09 PM
org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
WARNING: Not retrying send for:tcp://{10, -68, 4, -37}:4003; Sender is
disconnected.
Apr 29, 2010 1:48:09 PM
org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
WARNING: Not retrying send for:tcp://{10, -68, 4, -37}:4003; Sender is
disconnected.
Apr 29, 2010 1:48:09 PM
org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
WARNING: Not retrying send for:tcp://{10, -68, 4, -37}:4003; Sender is
disconnected.
Apr 29, 2010 1:48:09 PM
org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
WARNING: Not retrying send for:tcp://{10, -68, 4, -37}:4003; Sender is
disconnected.
Apr 29, 2010 1:48:09 PM
org.apache.catalina.tribes.transport.nio.ParallelNioSender doLoop
WARNING: Not retrying send for:tcp://{10, -68, 4, -37}:4003; Sender is
disconnected.
Apr 29, 2010 1:48:16 PM org.apache.catalina.ha.tcp.SimpleTcpCluster
memberAdded
INFO: Replication member
added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, -68, 4,
-37}:4003,{10, -68, 4, -37},4003, alive=1017,id={-62 69 63 -35 78 -21 64 -15
-105 -100 127 -60 -62 115 -83 12 }, payload={}, command={}, domain={}, ]
