I need help from the clustering experts here on the mailing list. I have two
Tomcat instances (6.0.32) running on a Red Hat Enterprise server (hosted at
RackSpace). When attempting to start the first instance in the cluster, it
fails because the NioReceiver cannot start (it is unable to open a port between
4000 and 4099):
From catalina.out:
Aug 15, 2012 2:32:37 PM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal
performance in production environments was not found on the java.
library.path:
/opt/jdk1.6.0_24/jre/lib/amd64/server:/opt/jdk1.6.0_24/jre/lib/amd64:/opt/jdk1.6.0_24/jre/../lib/amd64:/usr/java/packages
/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
Aug 15, 2012 2:32:37 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 1398 ms
Aug 15, 2012 2:32:37 PM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
Aug 15, 2012 2:32:37 PM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.32
Aug 15, 2012 2:32:37 PM org.apache.catalina.ha.tcp.SimpleTcpCluster start
INFO: Cluster is about to start
Aug 15, 2012 2:32:40 PM org.apache.catalina.tribes.transport.ReceiverBase bind
INFO: Unable to bind server socket to:/192.168.100.225:4099 throwing error.
Aug 15, 2012 2:32:40 PM org.apache.catalina.tribes.transport.nio.NioReceiver
start
SEVERE: Unable to start cluster receiver
java.net.BindException: Cannot assign requested address
at sun.nio.ch.Net.bind(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
at
org.apache.catalina.tribes.transport.ReceiverBase.bind(ReceiverBase.java:205)
at
org.apache.catalina.tribes.transport.ReceiverBase.bind(ReceiverBase.java:217)
at
org.apache.catalina.tribes.transport.ReceiverBase.bind(ReceiverBase.java:217)
at
org.apache.catalina.tribes.transport.ReceiverBase.bind(ReceiverBase.java:217)
…
Aug 15, 2012 2:32:40 PM org.apache.catalina.ha.tcp.SimpleTcpCluster start
SEVERE: Unable to start cluster.
org.apache.catalina.tribes.ChannelException: java.net.BindException: Cannot
assign requested address; No faulty members identified.
at
org.apache.catalina.tribes.group.ChannelCoordinator.internalStart(ChannelCoordinator.java:169)
at
org.apache.catalina.tribes.group.ChannelCoordinator.start(ChannelCoordinator.java:97)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.start(ChannelInterceptorBase.java:149)
at
org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.start(MessageDispatchInterceptor.java:147)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.start(ChannelInterceptorBase.java:149)
…
Caused by: java.net.BindException: Cannot assign requested address
at sun.nio.ch.Net.bind(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
at
org.apache.catalina.tribes.transport.ReceiverBase.bind(ReceiverBase.java:205)
…
Aug 15, 2012 2:32:40 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 2807 ms
Aug 15, 2012 2:32:40 PM org.apache.catalina.core.StandardServer await
SEVERE: StandardServer.await: create[49352]:
java.net.BindException: Address already in use
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:383)
at java.net.ServerSocket.bind(ServerSocket.java:328)
at java.net.ServerSocket.<init>(ServerSocket.java:194)
at
org.apache.catalina.core.StandardServer.await(StandardServer.java:406)
at org.apache.catalina.startup.Catalina.await(Catalina.java:676)
…
Aug 15, 2012 2:32:41 PM org.apache.catalina.core.StandardService stop
INFO: Stopping service Catalina
Aug 15, 2012 2:32:41 PM org.apache.catalina.startup.Catalina stop
SEVERE: Catalina.stop
LifecycleException: Pipeline has not been started
at
org.apache.catalina.core.StandardPipeline.stop(StandardPipeline.java:257)
at org.apache.catalina.core.ContainerBase.stop(ContainerBase.java:1099)
at org.apache.catalina.core.StandardEngine.stop(StandardEngine.java:468)
at
org.apache.catalina.core.StandardService.stop(StandardService.java:604)
at org.apache.catalina.core.StandardServer.stop(StandardServer.java:788)
at org.apache.catalina.startup.Catalina.stop(Catalina.java:662)
at org.apache.catalina.startup.Catalina.start(Catalina.java:629)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Here is my cluster configuration from server.xml:
<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
channelSendOptions="8">
<Manager className="org.apache.catalina.ha.session.DeltaManager"
expireSessionsOnShutdown="false" notifyListenersOnReplication="true" />
<Channel className="org.apache.catalina.tribes.group.GroupChannel">
<Membership
className="org.apache.catalina.tribes.membership.McastService"
address="228.0.0.4" port="45564" frequency="500" dropTime="3000" />
<Sender
className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
<Transport
className="org.apache.catalina.tribes.transport.nio.PooledParallelSender" />
</Sender>
<Receiver
className="org.apache.catalina.tribes.transport.nio.NioReceiver" address="auto"
port="4000" autoBind="100" selectorTimeout="5000" maxThreads="6" />
<Interceptor
className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector" />
<Interceptor
className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"
/>
</Channel>
<Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
filter="" />
<Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"
/>
<ClusterListener
className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener" />
<ClusterListener
className="org.apache.catalina.ha.session.ClusterSessionListener" />
</Cluster>
I checked to make sure multicasting is enabled (ifconfig eth0):
eth0 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx
inet addr:192.168.200.225 Bcast:192.168.203.255 Mask:255.255.252.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1984991 errors:0 dropped:0 overruns:0 frame:0
TX packets:1941912 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:568159401 (541.8 MiB) TX bytes:569789016 (543.3 MiB)
Interrupt:11 Memory:f8000000-f8012800
Is there anything I need to add to /etc/hosts or /etc/hosts.allow? Maybe
RackSpace's default rules are somehow blocking access, even though there are no
explicit firewall rules blocking 4000-4099. I don't know.
I had this same cluster configuration up and running on my MacBook Pro without
any issues. But now I am at a loss. Can anyone point me in a direction to
investigate further?
Thank you!
Jeff
