Re: BackupManager start fails under heavy load

[Keiichi Fujino]

2013/6/28 Patrick Savage 

> We have an issue in our Tomcat 7.0.30 clustered production environment on
> RHEL 5 where Tomcat fails to start our application when other nodes in the
> cluster are under extremely heavy load. It fails because the BackupManager
> cannot start the replicated map due to timeouts trying to connect to all
> the
> other nodes. The only way to recover from this seems to be shutting down
> almost all of the nodes and then starting them again. The cluster has 9
> nodes, but we have also had the problem with 6 nodes.
>
>
>
> Is there a way to ensure the application will start even if the
> BackupManager cannot connect to the other nodes?


No.
If replication map fails to start, associated context will fail to start.

I will implement a feature  to ensure the application will start even If
replication map fails to start.

-- 
Keiichi.Fujino

Re: BackupManager start fails under heavy load

[Vince Stewart]

Hi Patrick,
A similar problem has been reported before:
http://tomcat.10.n6.nabble.com/org-apache-catalina-tribes-ChannelException-Operation-has-timed-out-3000-ms-Faulty-members-tcp-64-88-td4656393.html
The important error message from your log output is:

   

>   Caused by: org.apache.catalina.tribes.ChannelException: Operation has
> timed out(3000 ms.).; Faulty members:tcp://{10, 230, 20, 86}:4001;
> tcp://{10, 230, 20, 87}:4001; tcp://{10, 230, 20, 94}:4001; tcp://{10, 230,
> 20, 95}:4001; tcp://{10, 230, 20, 70}:4001; tcp://{10, 230, 20, 89}:4001;
>
> at
>
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(Paral
> lelNioSender.java:109)
> ...
>

I am familiar with the code that generates this message; the problem is
that the sending operation is abandoned for any sender object which has not
been drained of data within timeout milliseconds. The "timeout" parameter
is declared in AbstractSender class as (long) 3000. By my reckoning a small
change to the timeout value could result a large reduction in messaging
failures.

According to information from this page:
http://tomcat.apache.org/tomcat-7.0-doc/config/cluster-sender.html

you should be able to increase the timeout parameter by setting a transport
attribute thus:

  

  

However, I can not find the code where the system reads the configuration
to override the default value; if you make the alteration and find the
error message still reports "3000ms", this would indicate an oversight in
the coding which could be reported.

BTW, your configuration for receiver has
selectorTimeout="100"

The code suggests that this should be the same value as sender/transport
timeout (ie 3000). The documentation says the default is 5000. My
examination of the code suggests that the PooledParallelSender class does
not read the configuration but always uses 5000. Nevertheless, you could
try setting that value to 5000 and seeing what happens.

BTW my own interest was to implement tribes at Internet connection speed;
by manipulating the parameter in question, my system copes with data
transfers that take multiple seconds.
http://tomcat.10.x6.nabble.com/overcoming-a-message-size-limitation-in-tribes-parallel-messaging-with-NioSender-tt4995446.html

BackupManager start fails under heavy load

[Patrick Savage]

We have an issue in our Tomcat 7.0.30 clustered production environment on
RHEL 5 where Tomcat fails to start our application when other nodes in the
cluster are under extremely heavy load. It fails because the BackupManager
cannot start the replicated map due to timeouts trying to connect to all the
other nodes. The only way to recover from this seems to be shutting down
almost all of the nodes and then starting them again. The cluster has 9
nodes, but we have also had the problem with 6 nodes.

 

Is there a way to ensure the application will start even if the
BackupManager cannot connect to the other nodes? Or is there some
configuration we can change to prevent the connection failure? E.g. should
we set some of the attributes on  such as timeout,
maxRetryAttempts, poolSize, or maxWait?

 

We see many "WARNING: Channel key is registered, but has had no interest ops
for the last 3000 ms." messages when we we have relatively heavy load.
Should we increase maxThreads on  to avoid this message and
potentially prevent the connection failures? 

 

This is the server.xml:

 

  





  

  

  



  

  

  

  



 





 





  

 

-

 

This is log for the tcp://{10, 230, 20, 85} node that failed to startup
because it timed out connecting to 6 of the 8 other nodes:

 

  Jun 26, 2013 5:29:01 PM
org.apache.catalina.tribes.tipis.AbstractReplicatedMap init

  WARNING: Unable to send map start message.

  Jun 26, 2013 5:29:01 PM org.apache.catalina.ha.session.BackupManager
startInternal

  SEVERE: Unable to start BackupManager: [/appv5]

  java.lang.RuntimeException: Unable to start replicated map.

at
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.init(AbstractReplicat
edMap.java:234)

at
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.(AbstractReplic
atedMap.java:176)

at
org.apache.catalina.tribes.tipis.LazyReplicatedMap.(LazyReplicatedMap.
java:104)

at
org.apache.catalina.ha.session.BackupManager.startInternal(BackupManager.jav
a:163)

at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)

at
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:
5294)

at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)

at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:9
01)

at
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)

at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:618)

at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:963)

at
org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1600)

at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

at java.util.concurrent.FutureTask.run(FutureTask.java:138)

at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:886)

at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
08)

at java.lang.Thread.run(Thread.java:662)

  Caused by: org.apache.catalina.tribes.ChannelException: Operation has
timed out(3000 ms.).; Faulty members:tcp://{10, 230, 20, 86}:4001;
tcp://{10, 230, 20, 87}:4001; tcp://{10, 230, 20, 94}:4001; tcp://{10, 230,
20, 95}:4001; tcp://{10, 230, 20, 70}:4001; tcp://{10, 230, 20, 89}:4001; 

at
org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(Paral
lelNioSender.java:109)

at
org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(Po
oledParallelSender.java:54)

at
org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(Repl
icationTransmitter.java:79)

at
org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoord
inator.java:78)

at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelI
nterceptorBase.java:79)

at
org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor.sendMess
age(ThroughputInterceptor.java:64)

at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelI
nterceptorBase.java:79)

at
org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sen
dMessage(MessageDispatchInterceptor.java:77)

at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelI
nterceptorBase.java:79)

at
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage
(TcpFailureDetector.java:89)

at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelI
nterceptorBase.java:79)

at
org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:224)

at
org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:182)

at org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:96)

at
org.apache.catalina.tribes.tipis.Abs