thanks for your reply.
first the fix you mentioned is not in the current tomcat release (6.0.16).
Since I have to build/patch the tomcat for a productive environment what
whould you recommend: patch the 6.0.16 sources or build tomcat from the
newest trunk?
We've ran another test and encountered a network error (packet got no ACK)
on the multicast interfaces. The wireshark log (part of) is attached below.
Perhaps you've an idea why this is happening!?
Thanks for your help!
Guido
--------------------- wireshark log start --------------------
No. Time Source Destination Protocol
Info
37433 1543.747634 223.99.114.20 223.99.114.21 TCP
36615 > 14022 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=568786762 TSER=0 WS=2
Frame 37433 (74 bytes on wire, 74 bytes captured)
Arrival Time: Feb 12, 2008 22:04:28.652917000
[Time delta from previous captured frame: 0.071229000 seconds]
[Time delta from previous displayed frame: 1543.747634000 seconds]
[Time since reference or first frame: 1543.747634000 seconds]
Frame Number: 37433
Frame Length: 74 bytes
Capture Length: 74 bytes
[Frame is marked: False]
[Protocols in frame: eth:ip:tcp]
[Coloring Rule Name: TCP SYN/FIN]
[Coloring Rule String: tcp.flags & 0x02 || tcp.flags.fin == 1]
Ethernet II, Src: Broadcom_11:2a:c1 (00:10:18:11:2a:c1), Dst:
Broadcom_17:7f:8c (00:10:18:17:7f:8c)
Destination: Broadcom_17:7f:8c (00:10:18:17:7f:8c)
Address: Broadcom_17:7f:8c (00:10:18:17:7f:8c)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
.... ..0. .... .... .... .... = LG bit: Globally unique address
(factory default)
Source: Broadcom_11:2a:c1 (00:10:18:11:2a:c1)
Address: Broadcom_11:2a:c1 (00:10:18:11:2a:c1)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
.... ..0. .... .... .... .... = LG bit: Globally unique address
(factory default)
Type: IP (0x0800)
Internet Protocol, Src: 223.99.114.20 (223.99.114.20), Dst: 223.99.114.21
(223.99.114.21)
Version: 4
Header length: 20 bytes
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
0000 00.. = Differentiated Services Codepoint: Default (0x00)
.... ..0. = ECN-Capable Transport (ECT): 0
.... ...0 = ECN-CE: 0
Total Length: 60
Identification: 0x926a (37482)
Flags: 0x04 (Don't Fragment)
0... = Reserved bit: Not set
.1.. = Don't fragment: Set
..0. = More fragments: Not set
Fragment offset: 0
Time to live: 64
Protocol: TCP (0x06)
Header checksum: 0x0561 [correct]
[Good: True]
[Bad : False]
Source: 223.99.114.20 (223.99.114.20)
Destination: 223.99.114.21 (223.99.114.21)
Transmission Control Protocol, Src Port: 36615 (36615), Dst Port: 14022
(14022), Seq: 0, Len: 0
Source port: 36615 (36615)
Destination port: 14022 (14022)
Sequence number: 0 (relative sequence number)
Header length: 40 bytes
Flags: 0x02 (SYN)
0... .... = Congestion Window Reduced (CWR): Not set
.0.. .... = ECN-Echo: Not set
..0. .... = Urgent: Not set
...0 .... = Acknowledgment: Not set
.... 0... = Push: Not set
.... .0.. = Reset: Not set
.... ..1. = Syn: Set
.... ...0 = Fin: Not set
Window size: 5840
Checksum: 0x7df2 [correct]
[Good Checksum: True]
[Bad Checksum: False]
Options: (20 bytes)
Maximum segment size: 1460 bytes
SACK permitted
Timestamps: TSval 568786762, TSecr 0
NOP
Window scale: 2 (multiply by 4)
No. Time Source Destination Protocol
Info
137308 1547.148341 223.99.114.20 223.99.114.21 TCP
36615 > 14022 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=568786762 TSER=0 WS=2
Frame 137308 (74 bytes on wire, 74 bytes captured)
Arrival Time: Feb 12, 2008 22:04:32.053624000
[Time delta from previous captured frame: 0.071236000 seconds]
[Time delta from previous displayed frame: 3.400707000 seconds]
[Time since reference or first frame: 1547.148341000 seconds]
Frame Number: 137308
Frame Length: 74 bytes
Capture Length: 74 bytes
[Frame is marked: False]
[Protocols in frame: eth:ip:tcp]
[Coloring Rule Name: TCP SYN/FIN]
[Coloring Rule String: tcp.flags & 0x02 || tcp.flags.fin == 1]
Ethernet II, Src: Broadcom_11:2a:c1 (00:10:18:11:2a:c1), Dst:
Broadcom_17:7f:8c (00:10:18:17:7f:8c)
Destination: Broadcom_17:7f:8c (00:10:18:17:7f:8c)
Address: Broadcom_17:7f:8c (00:10:18:17:7f:8c)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
.... ..0. .... .... .... .... = LG bit: Globally unique address
(factory default)
Source: Broadcom_11:2a:c1 (00:10:18:11:2a:c1)
Address: Broadcom_11:2a:c1 (00:10:18:11:2a:c1)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
.... ..0. .... .... .... .... = LG bit: Globally unique address
(factory default)
Type: IP (0x0800)
Internet Protocol, Src: 223.99.114.20 (223.99.114.20), Dst: 223.99.114.21
(223.99.114.21)
Version: 4
Header length: 20 bytes
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
0000 00.. = Differentiated Services Codepoint: Default (0x00)
.... ..0. = ECN-Capable Transport (ECT): 0
.... ...0 = ECN-CE: 0
Total Length: 60
Identification: 0x926a (37482)
Flags: 0x04 (Don't Fragment)
0... = Reserved bit: Not set
.1.. = Don't fragment: Set
..0. = More fragments: Not set
Fragment offset: 0
Time to live: 64
Protocol: TCP (0x06)
Header checksum: 0x0561 [correct]
[Good: True]
[Bad : False]
Source: 223.99.114.20 (223.99.114.20)
Destination: 223.99.114.21 (223.99.114.21)
Transmission Control Protocol, Src Port: 36615 (36615), Dst Port: 14022
(14022), Seq: 0, Len: 0
Source port: 36615 (36615)
Destination port: 14022 (14022)
Sequence number: 0 (relative sequence number)
Header length: 40 bytes
Flags: 0x02 (SYN)
0... .... = Congestion Window Reduced (CWR): Not set
.0.. .... = ECN-Echo: Not set
..0. .... = Urgent: Not set
...0 .... = Acknowledgment: Not set
.... 0... = Push: Not set
.... .0.. = Reset: Not set
.... ..1. = Syn: Set
.... ...0 = Fin: Not set
Window size: 5840
Checksum: 0x7df2 [correct]
[Good Checksum: True]
[Bad Checksum: False]
Options: (20 bytes)
Maximum segment size: 1460 bytes
SACK permitted
Timestamps: TSval 568786762, TSecr 0
NOP
Window scale: 2 (multiply by 4)
No. Time Source Destination Protocol
Info
37451 1544.749177 223.99.114.20 223.99.114.21 TCP
36616 > 14022 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=568787763 TSER=0 WS=2
Frame 37451 (74 bytes on wire, 74 bytes captured)
Arrival Time: Feb 12, 2008 22:04:29.654460000
[Time delta from previous captured frame: 0.076938000 seconds]
[Time delta from previous displayed frame: 1544.749177000 seconds]
[Time since reference or first frame: 1544.749177000 seconds]
Frame Number: 37451
Frame Length: 74 bytes
Capture Length: 74 bytes
[Frame is marked: False]
[Protocols in frame: eth:ip:tcp]
[Coloring Rule Name: TCP SYN/FIN]
[Coloring Rule String: tcp.flags & 0x02 || tcp.flags.fin == 1]
Ethernet II, Src: Broadcom_11:2a:c1 (00:10:18:11:2a:c1), Dst:
Broadcom_17:7f:8c (00:10:18:17:7f:8c)
Destination: Broadcom_17:7f:8c (00:10:18:17:7f:8c)
Address: Broadcom_17:7f:8c (00:10:18:17:7f:8c)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
.... ..0. .... .... .... .... = LG bit: Globally unique address
(factory default)
Source: Broadcom_11:2a:c1 (00:10:18:11:2a:c1)
Address: Broadcom_11:2a:c1 (00:10:18:11:2a:c1)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
.... ..0. .... .... .... .... = LG bit: Globally unique address
(factory default)
Type: IP (0x0800)
Internet Protocol, Src: 223.99.114.20 (223.99.114.20), Dst: 223.99.114.21
(223.99.114.21)
Version: 4
Header length: 20 bytes
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
0000 00.. = Differentiated Services Codepoint: Default (0x00)
.... ..0. = ECN-Capable Transport (ECT): 0
.... ...0 = ECN-CE: 0
Total Length: 60
Identification: 0xbb3b (47931)
Flags: 0x04 (Don't Fragment)
0... = Reserved bit: Not set
.1.. = Don't fragment: Set
..0. = More fragments: Not set
Fragment offset: 0
Time to live: 64
Protocol: TCP (0x06)
Header checksum: 0xdc8f [correct]
[Good: True]
[Bad : False]
Source: 223.99.114.20 (223.99.114.20)
Destination: 223.99.114.21 (223.99.114.21)
Transmission Control Protocol, Src Port: 36616 (36616), Dst Port: 14022
(14022), Seq: 0, Len: 0
Source port: 36616 (36616)
Destination port: 14022 (14022)
Sequence number: 0 (relative sequence number)
Header length: 40 bytes
Flags: 0x02 (SYN)
0... .... = Congestion Window Reduced (CWR): Not set
.0.. .... = ECN-Echo: Not set
..0. .... = Urgent: Not set
...0 .... = Acknowledgment: Not set
.... 0... = Push: Not set
.... .0.. = Reset: Not set
.... ..1. = Syn: Set
.... ...0 = Fin: Not set
Window size: 5840
Checksum: 0x2e75 [correct]
[Good Checksum: True]
[Bad Checksum: False]
Options: (20 bytes)
Maximum segment size: 1460 bytes
SACK permitted
Timestamps: TSval 568787763, TSecr 0
NOP
Window scale: 2 (multiply by 4)
No. Time Source Destination Protocol
Info
137326 1548.149884 223.99.114.20 223.99.114.21 TCP
36616 > 14022 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=568787763 TSER=0 WS=2
Frame 137326 (74 bytes on wire, 74 bytes captured)
Arrival Time: Feb 12, 2008 22:04:33.055167000
[Time delta from previous captured frame: 0.077082000 seconds]
[Time delta from previous displayed frame: 3.400707000 seconds]
[Time since reference or first frame: 1548.149884000 seconds]
Frame Number: 137326
Frame Length: 74 bytes
Capture Length: 74 bytes
[Frame is marked: False]
[Protocols in frame: eth:ip:tcp]
[Coloring Rule Name: TCP SYN/FIN]
[Coloring Rule String: tcp.flags & 0x02 || tcp.flags.fin == 1]
Ethernet II, Src: Broadcom_11:2a:c1 (00:10:18:11:2a:c1), Dst:
Broadcom_17:7f:8c (00:10:18:17:7f:8c)
Destination: Broadcom_17:7f:8c (00:10:18:17:7f:8c)
Address: Broadcom_17:7f:8c (00:10:18:17:7f:8c)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
.... ..0. .... .... .... .... = LG bit: Globally unique address
(factory default)
Source: Broadcom_11:2a:c1 (00:10:18:11:2a:c1)
Address: Broadcom_11:2a:c1 (00:10:18:11:2a:c1)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
.... ..0. .... .... .... .... = LG bit: Globally unique address
(factory default)
Type: IP (0x0800)
Internet Protocol, Src: 223.99.114.20 (223.99.114.20), Dst: 223.99.114.21
(223.99.114.21)
Version: 4
Header length: 20 bytes
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
0000 00.. = Differentiated Services Codepoint: Default (0x00)
.... ..0. = ECN-Capable Transport (ECT): 0
.... ...0 = ECN-CE: 0
Total Length: 60
Identification: 0xbb3b (47931)
Flags: 0x04 (Don't Fragment)
0... = Reserved bit: Not set
.1.. = Don't fragment: Set
..0. = More fragments: Not set
Fragment offset: 0
Time to live: 64
Protocol: TCP (0x06)
Header checksum: 0xdc8f [correct]
[Good: True]
[Bad : False]
Source: 223.99.114.20 (223.99.114.20)
Destination: 223.99.114.21 (223.99.114.21)
Transmission Control Protocol, Src Port: 36616 (36616), Dst Port: 14022
(14022), Seq: 0, Len: 0
Source port: 36616 (36616)
Destination port: 14022 (14022)
Sequence number: 0 (relative sequence number)
Header length: 40 bytes
Flags: 0x02 (SYN)
0... .... = Congestion Window Reduced (CWR): Not set
.0.. .... = ECN-Echo: Not set
..0. .... = Urgent: Not set
...0 .... = Acknowledgment: Not set
.... 0... = Push: Not set
.... .0.. = Reset: Not set
.... ..1. = Syn: Set
.... ...0 = Fin: Not set
Window size: 5840
Checksum: 0x2e75 [correct]
[Good Checksum: True]
[Bad Checksum: False]
Options: (20 bytes)
Maximum segment size: 1460 bytes
SACK permitted
Timestamps: TSval 568787763, TSecr 0
NOP
Window scale: 2 (multiply by 4)
--------------------- wireshark log end --------------------
Filip Hanik - Dev Lists wrote:
>
> [CLUSTERLOG] NioReceiver - WARN - 02/12/2008 17:34:23,895 - NioReceiver -
> Channel key is registered, but has had no interest ops for the last 3000
> ms.
> (canc
> elled:false):[EMAIL PROTECTED] last access:2008-02-12
> 17:34:17.957
> and finally:
>
> this message most likely indicates that the thread pool handling
> incoming request is busy.
> at this point do a thread dump to see the state of the threads.
> There was a deadlock fix in relation to Raul's error, and was fixed in
> this SVN commit
> http://svn.apache.org/viewvc?view=rev&revision=618823
>
> it has been proposed to be merged back into the 6.0.x branch
>
> Filip
>
> gp456 wrote:
>> Hello,
>>
>> we have a similar problem as Raul Garcia has posted in
>> http://www.nabble.com/Tomcat-6---Cluster-error.-to14921385.html#a14921385
>> this message, but we're receiving the error of a disappearing member
>> after
>> appr. 12 hours!!
>> Our load is also similar: 1 message/sec.
>>
>> We are using a hardware loadbalancer in front of two apache/tomcat pairs.
>> The apache are connected via ajp
>> to the local tomcat or (if the local tomcat is down) to the tomcat on the
>> other node.
>> Our config:
>> Node1/2:
>> Apache 2.0.59
>> Tomcat 6.0.13
>>
>> Tomcat conf Node 1:
>>
>> ...
>> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
>> channelSendOptions="8">
>>
>> <Manager
>> className="org.apache.catalina.ha.session.DeltaManager"
>> expireSessionsOnShutdown="false"
>> notifyListenersOnReplication="true"/>
>>
>> <Channel
>> className="org.apache.catalina.tribes.group.GroupChannel">
>> <Membership
>> className="org.apache.catalina.tribes.membership.McastService"
>> address="239.252.100.12"
>> port="14022"
>> frequency="1000"
>> dropTime="30000"/>
>> <Receiver
>> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
>> address="223.99.114.21"
>> port="14022"
>> autoBind="100"
>> selectorTimeout="5000"
>> maxThreads="6"/>
>>
>> <Sender
>> className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
>> <Transport
>> className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
>> timeout="60000" maxRetryAttempts="10" keepAliveCount="0" />
>> </Sender>
>> <Interceptor
>> className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
>> <Interceptor
>> className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
>> <Interceptor
>> className="org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor"
>> interval="1000" />
>> </Channel>
>>
>> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
>> filter=""/>
>> <Valve
>> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
>>
>> <!-- Deployer
>> className="org.apache.catalina.ha.deploy.FarmWarDeployer"
>> tempDir="/tmp/war-temp/"
>> deployDir="/tmp/war-deploy/"
>> watchDir="/tmp/war-listen/"
>> watchEnabled="false"/ -->
>>
>> <ClusterListener
>> className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
>> <ClusterListener
>> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
>> </Cluster>
>> ...
>>
>> Tomcat conf Node 2:
>> ...
>> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
>> channelSendOptions="8">
>>
>> <Manager
>> className="org.apache.catalina.ha.session.DeltaManager"
>> expireSessionsOnShutdown="false"
>> notifyListenersOnReplication="true"/>
>>
>> <Channel
>> className="org.apache.catalina.tribes.group.GroupChannel">
>> <Membership
>> className="org.apache.catalina.tribes.membership.McastService"
>> address="239.252.100.12"
>> port="14022"
>> frequency="1000"
>> dropTime="30000"/>
>> <Receiver
>> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
>> address="223.99.114.20"
>> port="14022"
>> autoBind="100"
>> selectorTimeout="5000"
>> maxThreads="6"/>
>>
>> <Sender
>> className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
>> <Transport
>> className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
>> timeout="60000" maxRetryAttempts="10" keepAliveCount="0" />
>> </Sender>
>> <Interceptor
>> className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
>> <Interceptor
>> className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
>> <Interceptor
>> className="org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor"
>> interval="1000" />
>> </Channel>
>>
>> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
>> filter=""/>
>> <Valve
>> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
>>
>> <!-- Deployer
>> className="org.apache.catalina.ha.deploy.FarmWarDeployer"
>> tempDir="/tmp/war-temp/"
>> deployDir="/tmp/war-deploy/"
>> watchDir="/tmp/war-listen/"
>> watchEnabled="false"/ -->
>>
>> <ClusterListener
>> className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
>> <ClusterListener
>> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
>> </Cluster>
>> ...
>>
>> Log On Node 1:
>>
>> ...
>> [CLUSTERLOG] main - INFO - 02/12/2008 10:05:15,875 - McastService -
>> Sleeping
>> for 2000 milliseconds to establish cluster membership, start level:8
>> [CLUSTERLOG] pool-2-thread-1 - INFO - 02/12/2008 10:05:15,908 -
>> BufferPool -
>> Created a buffer pool with max size:104857600 bytes of
>> type:org.apache.catalina
>> .tribes.io.BufferPool15Impl
>> [CLUSTERLOG] main - INFO - 02/12/2008 10:05:17,877 - McastService - Done
>> sleeping, membership established, start level:8
>> [CLUSTERLOG] main - INFO - 02/12/2008 10:05:17,884 - JvmRouteBinderValve
>> -
>> JvmRouteBinderValve started
>> [CLUSTERLOG] pool-2-thread-3 - WARN - 02/12/2008 10:05:19,328 -
>> ClusterListener - Context manager doesn't exist:
>> [CLUSTERLOG] main - INFO - 02/12/2008 10:05:20,015 - DeltaManager -
>> Register
>> manager to cluster element Host with name localhost
>> [CLUSTERLOG] main - INFO - 02/12/2008 10:05:20,015 - DeltaManager -
>> Starting
>> clustering manager at
>> [CLUSTERLOG] main - WARN - 02/12/2008 10:05:20,026 - DeltaManager -
>> Manager
>> [], requesting session state from
>> org.apache.catalina.tribes.membership.MemberIm
>> pl[tcp://223.99.114.21:14022,223.99.114.21,14022, alive=167423,id={10 -86
>> 53
>> 5 6 -125 71 -7 -109 5 -36 4 48 -46 45 65 }, payload={}, command={},
>> domain={},
>> ]. This operation will timeout if no session state has been received
>> within
>> 60 seconds.
>> [CLUSTERLOG] pool-1-thread-1 - INFO - 02/12/2008 10:05:20,045 -
>> ThroughputInterceptor - ThroughputInterceptor Report[
>> Tx Msg:1 messages
>> Sent:0,00 MB (total)
>> Sent:0,00 MB (application)
>> Time:0,02 seconds
>> Tx Speed:0,03 MB/sec (total)
>> TxSpeed:0,03 MB/sec (application)
>> Error Msg:0
>> Rx Msg:2 messages
>> Rx Speed:0,00 MB/sec (since 1st msg)
>> Received:0,00 MB]
>> ...
>> after some time we're getting this entry:
>>
>> [CLUSTERLOG] pool-2-thread-4 - WARN - 02/12/2008 15:00:52,044 -
>> ClusterListener - Context manager doesn't exist:
>> [CLUSTERLOG] pool-2-thread-5 - WARN - 02/12/2008 15:00:52,910 -
>> ClusterListener - Context manager doesn't exist:
>> [CLUSTERLOG] pool-2-thread-2 - WARN - 02/12/2008 15:00:54,299 -
>> ClusterListener - Context manager doesn't exist:
>> [CLUSTERLOG] pool-2-thread-6 - WARN - 02/12/2008 15:00:54,707 -
>> ClusterListener - Context manager doesn't exist:
>> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] -
>> INFO -
>> 02/12/2008 15:00:56,011 - DeltaManager - Register manager to cluster
>> element H
>> ost with name localhost
>> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] -
>> INFO -
>> 02/12/2008 15:00:56,011 - DeltaManager - Starting clustering manager at
>> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] -
>> WARN -
>> 02/12/2008 15:00:56,012 - DeltaManager - Manager [], requesting session
>> state f
>> rom
>> org.apache.catalina.tribes.membership.MemberImpl[tcp://223.99.114.21:14022,223.99.114.21,14022,
>> alive=17902419,id={10 -86 53 5 6 -125 71 -7 -109 5 -36 4
>> 48 -46 45 65 }, payload={}, command={}, domain={}, ]. This operation
>> will
>> timeout if no session state has been received within 60 seconds.
>> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] -
>> INFO -
>> 02/12/2008 15:01:02,245 - DeltaManager - Manager []; session state send
>> at
>> 12.0
>> 2.08 15:00 received in 6.233 ms.
>> [CLUSTERLOG] pool-2-thread-2 - INFO - 02/12/2008 15:01:13,178 -
>> ThroughputInterceptor - ThroughputInterceptor Report[
>> Tx Msg:20003 messages
>> Sent:143,22 MB (total)
>> Sent:143,22 MB (application)
>> Time:54,66 seconds
>> Tx Speed:2,62 MB/sec (total)
>> TxSpeed:2,62 MB/sec (application)
>> Error Msg:0
>> Rx Msg:20000 messages
>> Rx Speed:0,01 MB/sec (since 1st msg)
>> Received:143,04 MB]
>>
>> [CLUSTERLOG] pool-2-thread-5 - INFO - 02/12/2008 15:23:49,278 -
>> ThroughputInterceptor - ThroughputInterceptor Report[
>> Tx Msg:20606 messages
>> Sent:156,13 MB (total)
>> Sent:156,13 MB (application)
>> Time:61,55 seconds
>> Tx Speed:2,54 MB/sec (total)
>> TxSpeed:2,54 MB/sec (application)
>> Error Msg:9
>> Rx Msg:21000 messages
>> Rx Speed:0,01 MB/sec (since 1st msg)
>> Received:150,14 MB]
>>
>> and now we're receiving and sending message but also Error messages....
>> and then:
>>
>> [CLUSTERLOG] NioReceiver - WARN - 02/12/2008 17:34:23,895 - NioReceiver -
>> Channel key is registered, but has had no interest ops for the last 3000
>> ms.
>> (canc
>> elled:false):[EMAIL PROTECTED] last access:2008-02-12
>> 17:34:17.957
>> and finally:
>>
>> [CLUSTERLOG] 4564 - INFO - 02/12/2008 22:04:28,652 - TcpFailureDetector -
>> Received
>> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://
>> 223.99.114.21:14022,223.99.114.21,14022, alive=43315450,id={10 -86 53 5 6
>> -125 71 -7 -109 5 -36 4 48 -46 45 65 }, payload={}, command={},
>> domain={},
>> ]] mess
>> age. Will verify.
>> [CLUSTERLOG] 4564 - INFO - 02/12/2008 22:04:29,654 - TcpFailureDetector -
>> Verification complete. Member
>> disappeared[org.apache.catalina.tribes.membership.Me
>> mberImpl[tcp://223.99.114.21:14022,223.99.114.21,14022,
>> alive=43316452,id={10 -86 53 5 6 -125 71 -7 -109 5 -36 4 48 -46 45 65 },
>> payload={}, command={}, dom
>> ain={}, ]]
>> [CLUSTERLOG] 4564 - INFO - 02/12/2008 22:04:29,654 - SimpleTcpCluster -
>> Received member
>> disappeared:org.apache.catalina.tribes.membership.MemberImpl[tcp://2
>> 23.99.114.21:14022,223.99.114.21,14022, alive=43316452,id={10 -86 53 5 6
>> -125 71 -7 -109 5 -36 4 48 -46 45 65 }, payload={}, command={},
>> domain={}, ]
>> [CLUSTERLOG] 4564 - ERROR - 02/12/2008 22:04:29,654 - SimpleTcpCluster -
>> Unable to send message through cluster sender.
>> org.apache.catalina.tribes.ChannelException: Operation has timed
>> out(60000
>> ms.).; Faulty members:tcp://223.99.114.21:14022;
>> at
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:97)
>> at
>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
>> at
>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80)
>> at
>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78)
>> at
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>> at
>> org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor.sendMessage(ThroughputInterceptor.java:61)
>> at
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>> at
>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:60)
>> at
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>> at
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87)
>> at
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
>> at
>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
>> at
>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
>> at
>> org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:835)
>> at
>> org.apache.catalina.ha.tcp.SimpleTcpCluster.sendClusterDomain(SimpleTcpCluster.java:814)
>> at
>> org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:551)
>> at
>> org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:535)
>> at
>> org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:517)
>> at
>> org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:428)
>> at
>> org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:362)
>> at
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
>> at
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>> at
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:261)
>> at
>> org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
>> at
>> org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283)
>> at
>> org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)
>> at
>> org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
>> at
>> org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
>> at
>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:686)
>> at java.lang.Thread.run(Thread.java:595)
>> [CLUSTERLOG] 4575 - ERROR - 02/12/2008 22:04:34,920 - SimpleTcpCluster -
>> Unable to send message through cluster sender.
>> org.apache.catalina.tribes.ChannelException: No destination given; No
>> faulty
>> members identified.
>> at
>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:194)
>> at
>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
>> ...
>> the error messages are repeating...
>>
>>
>> Log On Node 2:
>>
>> [CLUSTERLOG] main - WARN - 02/12/2008 10:02:35,175 - SimpleTcpCluster -
>> Dynamic setProperty(className,value) has been disabled, please use
>> explicit
>> properties
>> for the element you are trying to identify
>> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:35,954 - SimpleTcpCluster -
>> Cluster is about to start
>> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:35,973 - ReceiverBase -
>> Receiver
>> Server Socket bound to:/223.99.114.21:14022
>> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:36,025 - McastService -
>> Setting
>> cluster mcast soTimeout to 1000
>> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:36,037 - McastService -
>> Sleeping
>> for 2000 milliseconds to establish cluster membership, start level:4
>> [CLUSTERLOG] Thread-3 - INFO - 02/12/2008 10:02:36,913 - SimpleTcpCluster
>> -
>> Replication member
>> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://22
>> 3.99.114.20:14022,223.99.114.20,14022, alive=322404622,id={109 -10 62 -21
>> 19
>> 102 69 -42 -92 -123 -59 -15 37 16 50 -5 }, payload={}, command={},
>> domain={}, ]
>> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:38,039 - McastService - Done
>> sleeping, membership established, start level:4
>> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:38,044 - McastService -
>> Sleeping
>> for 2000 milliseconds to establish cluster membership, start level:8
>> [CLUSTERLOG] pool-2-thread-1 - INFO - 02/12/2008 10:02:38,074 -
>> BufferPool -
>> Created a buffer pool with max size:104857600 bytes of
>> type:org.apache.catalina.t
>> ribes.io.BufferPool15Impl
>> [CLUSTERLOG] pool-2-thread-3 - WARN - 02/12/2008 10:02:38,312 -
>> ClusterListener - Context manager doesn't exist:
>> [CLUSTERLOG] pool-2-thread-4 - WARN - 02/12/2008 10:02:38,816 -
>> ClusterListener - Context manager doesn't exist:
>> [CLUSTERLOG] pool-2-thread-5 - WARN - 02/12/2008 10:02:39,277 -
>> ClusterListener - Context manager doesn't exist:
>> [CLUSTERLOG] pool-2-thread-6 - WARN - 02/12/2008 10:02:39,337 -
>> ClusterListener - Context manager doesn't exist:
>> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:40,046 - McastService - Done
>> sleeping, membership established, start level:8
>> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:40,053 - JvmRouteBinderValve
>> -
>> JvmRouteBinderValve started
>> [CLUSTERLOG] pool-2-thread-1 - WARN - 02/12/2008 10:02:42,816 -
>> ClusterListener - Context manager doesn't exist:
>> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:43,093 - DeltaManager -
>> Register
>> manager to cluster element Host with name localhost
>> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:43,093 - DeltaManager -
>> Starting
>> clustering manager at
>> [CLUSTERLOG] main - WARN - 02/12/2008 10:02:43,100 - DeltaManager -
>> Manager
>> [], requesting session state from
>> org.apache.catalina.tribes.membership.MemberImpl
>> [tcp://223.99.114.20:14022,223.99.114.20,14022, alive=322410633,id={109
>> -10
>> 62 -21 19 102 69 -42 -92 -123 -59 -15 37 16 50 -5 }, payload={},
>> command={},
>> domai
>> n={}, ]. This operation will timeout if no session state has been
>> received
>> within 60 seconds.
>> [CLUSTERLOG] pool-1-thread-1 - INFO - 02/12/2008 10:02:43,112 -
>> ThroughputInterceptor - ThroughputInterceptor Report[
>> Tx Msg:1 messages
>> Sent:0,00 MB (total)
>> Sent:0,00 MB (application)
>> Time:0,01 seconds
>> Tx Speed:0,04 MB/sec (total)
>> TxSpeed:0,04 MB/sec (application)
>> Error Msg:0
>> Rx Msg:6 messages
>> Rx Speed:0,00 MB/sec (since 1st msg)
>> Received:0,01 MB]
>>
>> [CLUSTERLOG] main - INFO - 02/12/2008 10:02:43,719 - DeltaManager -
>> Manager
>> []; session state send at 12.02.08 10:02 received in 624 ms.
>> [CLUSTERLOG] Thread-31 - INFO - 02/12/2008 10:04:55,941 -
>> TcpFailureDetector
>> - Verification complete. Member
>> disappeared[org.apache.catalina.tribes.membership
>> .MemberImpl[tcp://223.99.114.20:14022,223.99.114.20,14022,
>> alive=322543666,id={109 -10 62 -21 19 102 69 -42 -92 -123 -59 -15 37 16
>> 50
>> -5 }, payload={}, comman
>> d={66 65 66 89 45 65 76 69 88 ...(9)}, domain={}, ]]
>> [CLUSTERLOG] Thread-30 - INFO - 02/12/2008 10:04:55,941 -
>> TcpFailureDetector
>> - Verification complete. Member
>> disappeared[org.apache.catalina.tribes.membership
>> .MemberImpl[tcp://223.99.114.20:14022,223.99.114.20,14022,
>> alive=322543666,id={109 -10 62 -21 19 102 69 -42 -92 -123 -59 -15 37 16
>> 50
>> -5 }, payload={}, comman
>> d={66 65 66 89 45 65 76 69 88 ...(9)}, domain={}, ]]
>> [CLUSTERLOG] Thread-31 - INFO - 02/12/2008 10:04:55,941 -
>> SimpleTcpCluster -
>> Received member
>> disappeared:org.apache.catalina.tribes.membership.MemberImpl[tcp:
>> //223.99.114.20:14022,223.99.114.20,14022, alive=322543666,id={109 -10 62
>> -21 19 102 69 -42 -92 -123 -59 -15 37 16 50 -5 }, payload={}, command={66
>> 65
>> 66 89 4
>> 5 65 76 69 88 ...(9)}, domain={}, ]
>> [CLUSTERLOG] Thread-30 - INFO - 02/12/2008 10:04:55,941 -
>> SimpleTcpCluster -
>> Received member
>> disappeared:org.apache.catalina.tribes.membership.MemberImpl[tcp:
>> //223.99.114.20:14022,223.99.114.20,14022, alive=322543666,id={109 -10 62
>> -21 19 102 69 -42 -92 -123 -59 -15 37 16 50 -5 }, payload={}, command={66
>> 65
>> 66 89 4
>> 5 65 76 69 88 ...(9)}, domain={}, ]
>> [CLUSTERLOG] Thread-36 - INFO - 02/12/2008 10:05:19,299 -
>> SimpleTcpCluster -
>> Replication member
>> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://2
>> 23.99.114.20:14022,223.99.114.20,14022, alive=2028,id={19 -115 -1 -101 8
>> 91
>> 66 -91 -112 105 -121 109 -82 11 56 50 }, payload={}, command={},
>> domain={},
>> ]
>> [CLUSTERLOG] pool-2-thread-4 - INFO - 02/12/2008 10:25:11,742 -
>> ThroughputInterceptor - ThroughputInterceptor Report[
>> Tx Msg:892 messages
>> Sent:5,82 MB (total)
>> Sent:5,82 MB (application)
>> Time:1,33 seconds
>> Tx Speed:4,39 MB/sec (total)
>> TxSpeed:4,39 MB/sec (application)
>> Error Msg:0
>> Rx Msg:1000 messages
>> Rx Speed:0,00 MB/sec (since 1st msg)
>> Received:4,47 MB]
>>
>> ...
>> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] -
>> INFO -
>> 02/12/2008 15:01:16,005 - DeltaManager - Manager [] expiring sessions
>> upon
>> shutdo
>> wn
>> [CLUSTERLOG] pool-2-thread-1 - WARN - 02/12/2008 15:01:20,014 -
>> ClusterListener - Context manager doesn't exist:
>> [CLUSTERLOG] pool-2-thread-1 - WARN - 02/12/2008 15:01:20,015 -
>> ClusterListener - Context manager doesn't exist:
>> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] -
>> INFO -
>> 02/12/2008 15:01:20,219 - DeltaManager - Register manager to cluster
>> element Hos
>> t with name localhost
>> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] -
>> INFO -
>> 02/12/2008 15:01:20,220 - DeltaManager - Starting clustering manager at
>> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] -
>> WARN -
>> 02/12/2008 15:01:20,220 - DeltaManager - Manager [], requesting session
>> state fro
>> m
>> org.apache.catalina.tribes.membership.MemberImpl[tcp://223.99.114.20:14022,223.99.114.20,14022,
>> alive=17762449,id={19 -115 -1 -101 8 91 66 -91 -112 105 -121
>> 109 -82 11 56 50 }, payload={}, command={}, domain={}, ]. This operation
>> will timeout if no session state has been received within 60 seconds.
>> [CLUSTERLOG] ContainerBackgroundProcessor[StandardEngine[Catalina]] -
>> INFO -
>> 02/12/2008 15:01:24,093 - DeltaManager - Manager []; session state send
>> at
>> 12.02.
>> 08 15:01 received in 3.872 ms.
>> [CLUSTERLOG] pool-1-thread-1 - INFO - 02/12/2008 15:22:41,143 -
>> ThroughputInterceptor - ThroughputInterceptor Report[
>> Tx Msg:21000 messages
>> Sent:150,30 MB (total)
>> Sent:150,30 MB (application)
>> Time:70,58 seconds
>> Tx Speed:2,13 MB/sec (total)
>> TxSpeed:2,13 MB/sec (application)
>> Error Msg:0
>> Rx Msg:20534 messages
>> Rx Speed:0,01 MB/sec (since 1st msg)
>> Received:156,92 MB]
>>
>> after this entry we're NOT receiving any messages (Rx Msg):
>>
>> [CLUSTERLOG] pool-1-thread-2 - INFO - 02/12/2008 15:40:02,612 -
>> ThroughputInterceptor - ThroughputInterceptor Report[
>> Tx Msg:22000 messages
>> Sent:154,75 MB (total)
>> Sent:154,75 MB (application)
>> Time:72,92 seconds
>> Tx Speed:2,12 MB/sec (total)
>> TxSpeed:2,12 MB/sec (application)
>> Error Msg:0
>> Rx Msg:20534 messages
>> Rx Speed:0,01 MB/sec (since 1st msg)
>> Received:156,92 MB]
>>
>> [CLUSTERLOG] pool-1-thread-2 - INFO - 02/12/2008 15:54:15,184 -
>> ThroughputInterceptor - ThroughputInterceptor Report[
>> Tx Msg:23022 messages
>> Sent:156,30 MB (total)
>> Sent:156,30 MB (application)
>> Time:74,46 seconds
>> Tx Speed:2,10 MB/sec (total)
>> TxSpeed:2,10 MB/sec (application)
>> Error Msg:0
>> Rx Msg:20534 messages
>> Rx Speed:0,01 MB/sec (since 1st msg)
>> Received:156,92 MB]
>> ...
>>
>>
>>
>> Many thanks for your help!!!
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
>
>
--
View this message in context:
http://www.nabble.com/Tomcat-Cluster-Error---Operation-has-timed-out-tp15457470p15610793.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
