I apologize for the wall of text. I'm working through the clustering and farm deployment documentation for Tomcat 6 / Tomcat 7. To that end I've set up a 3 node cluster load balanced by an Apache web server on a single system.
The environment and configuration particulars are at the end of the mail message. Problem summary: The application is a quick sample that does the following: 1. Form-based login 2. Some personal information that can be customized (session variable) 3. Random pet generated from a pet factory and a properties file (session variable) 4. Exit message on logout containing the chosen pet and the personalization info If I pre-deploy the distributable application to all three nodes, I get reasonable log entries, and the application works as expected. If I start up Tomcat and then deploy the application to the three nodes, I get error messages in the logs, and the application is not available until the last timeout for clustering has expired (default of 60 seconds). However, once this timeout has passed, clustering seems to work. I observe similar problems when using the farm deployer. The application does not seem to be available until it is deployed across the 3 nodes and the cluster timeout has passed. I have not tried these tests in a controlled fashion using Tomcat from subversion. So, how do I remove the errors from the logs? . . . . just my two cents. /mde/ Sequence of events which works cleanly: 1. Copy the RPets.war file to each of the three $CATALINA_BASE/webapps directories 2. Start up each Tomcat instance - wait 5 seconds between startups so ports can be chosen for clustering (I could configure these) 3. Test the application Log samples: 1. Cluster membership adds normally (logs from one node) INFO: Receiver Server Socket bound to:/127.0.0.1:4001 INFO: Replication member added: org.apache.catalina.tribes.membership.MemberImpl[tcp://{127, 0, 0, 1}:4000, {127, 0, 0, 1},4000, alive=6538,id={67 -53 75 3 -46 11 67 34 -67 -33 -126 -125 107 115 76 -17 }, payload={}, command={}, domain={}, ] INFO: Replication member added: org.apache.catalina.tribes.membership.MemberImpl[tcp://{127, 0, 0, 1}:4002, {127, 0, 0, 1},4002, alive=1013,id={62 -72 -97 73 10 97 74 -30 -109 -12 93 -125 114 -109 4 24 }, payload={}, command={}, domain={}, ] 2. Session replication information received normally (logs from one node) INFO: Deploying configuration descriptor RPets.xml INFO: Register manager /RPets to cluster element Host with name localhost INFO: Starting clustering manager at /RPets WARNING: Manager [/RPets], requesting session state from org.apache.catalina.tribes.membership.MemberImpl[tcp://{127, 0, 0, 1}:4000, {127, 0, 0, 1},4000, alive=9540, id={67 -53 75 3 -46 11 67 34 -67 -33 -126 -125 107 115 76 -17 }, payload={}, command={}, domain={}, ]. This operation will timeout if no session state has been received within 60 seconds. INFO: Manager [/RPets]; session state send at 8/7/10 11:30 AM received in 275 ms. Operation: The application works as expected. I can turn off the node that is being accessed, and after a brief pause the web browser will continue to display the results. All session variables are duplicated as expected. I do notice the following on shutdown: SEVERE: The web application [/RPets] appears to have started a thread named [pool-1-thread-1] but has failed to stop it. This is very likely to create a memory leak. This error is not present when running on a Tomcat instance without clustering enabled (web application is marked distributable, but no <Cluster> element is present). Sequence of events which causes the errors: 1. Start up each Tomcat instance - wait 5 seconds between startups so ports can be chosen for clustering (I could configure these) 2. Copy the RPets.war file to each of the three $CATALINA_BASE/webapps directories 3. Test the application Log samples: 1. Cluster membership adds normally (logs from one node) INFO: Replication member added: org.apache.catalina.tribes.membership.MemberImpl[tcp://{127, 0, 0, 1}:4000, {127, 0, 0, 1},4000, alive=6538,id={-33 84 107 57 -77 93 76 39 -85 92 -37 -74 71 26 38 -35 }, payload={}, command={}, domain={}, ] INFO: Replication member added: org.apache.catalina.tribes.membership.MemberImpl[tcp://{127, 0, 0, 1}:4002, {127, 0, 0, 1},4002, alive=1030,id={-121 38 109 -46 -48 66 77 -80 -70 -101 127 80 5 58 119 -70 }, payload={}, command={}, domain={}, ] 2. After copying the war file to $CATALINA_BASE/webapps (no delay to make sure application is deployed) INFO: Deploying web application archive RPets.war INFO: Register manager /RPets to cluster element Host with name localhost INFO: Starting clustering manager at /RPets WARNING: Manager [/RPets], requesting session state from org.apache.catalina.tribes.membership.MemberImpl[tcp://{127, 0, 0, 1}:4000, {127, 0, 0, 1},4000, alive=661430, id={-33 84 107 57 -77 93 76 39 -85 92 -37 -74 71 26 38 -35 }, payload={}, command={}, domain={}, ]. This operation will timeout if no session state has been received within 60 seconds. SEVERE: Manager [/RPets]: No session state send at 8/7/10 12:41 PM received, timing out after 60,034 ms. WARNING: Manager [/RPets]: Drop message SESSION-GET-ALL inside GET_ALL_SESSIONS sync phase start date 8/7/10 12:41 PM message date 12/31/69 4:00 PM (what is the message date of 12/31/69 4:00 PM ?) Operation: The application works as expected once the cluster timeout is reached. I can turn off the node that is being accessed, and after a brief pause the web browser will continue to display the results. All session variables are duplicated as expected. The particulars: 1. OS - Fedora 13 2.6.33.6-147.2.4.fc13.i686 a. firewall modified to permit multicast -A INPUT -d 224.0.0.0/4 -m state --state NEW -j ACCEPT b. multicast route added ip route add to multicast 224.0.0.0/4 dev eth0 c. host name added to 127.0.0.1 2. Java - JRE/JDK 1.6.0_21 from Sun/Oracle 3. Tomcat - 6.0.29 from tomcat.apache.org a. Using 1 $CATALINA_HOME and 3 $CATALINA_BASE instances b. libtcnative built, installed, and recognized 4. Distributable web application (RPets - for Random Pets) 5. Apache 2.2.15 (packaged by Fedora) 6. mod_jk 1.2.30 built from source 7. NetBeans 6.8 development environment (hence the path="/RPets" in context.xml) The configuration files: 1. Typical server.xml file without farm deployment. Each server.xml file has the following ports changed: shutdown, HTTP/1.1 and redirect, AJP and redirect. <?xml version='1.0' encoding='utf-8'?> <Server port="8015" shutdown="SHUTDOWN"> <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" /> <Listener className="org.apache.catalina.core.JasperListener" /> <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" /> <Listener className="org.apache.catalina.mbeans.ServerLifecycleListener" /> <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" /> <GlobalNamingResources> <Resource name="UserDatabase" auth="Container" type="org.apache.catalina.UserDatabase" description="User database that can be updated and saved" factory="org.apache.catalina.users.MemoryUserDatabaseFactory" pathname="conf/tomcat-users.xml" /> </GlobalNamingResources> <Service name="Catalina"> <Connector port="8090" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8453"/> <Connector port="8019" protocol="AJP/1.3" redirectPort="8453"/> <Engine name="Catalina" defaultHost="localhost" jvmRoute="deimos"> <Realm className="org.apache.catalina.realm.UserDatabaseRealm" resourceName="UserDatabase"/> <Host name="localhost" appBase="webapps" unpackWARs="true" autoDeploy="true" xmlValidation="false" xmlNamespaceAware="false"> <!-- cluster here since farm deployment doesn't work at the Engine level --> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/> <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" prefix="localhost_access_log." suffix=".txt" pattern="common" resolveHosts="false"/> </Host> </Engine> </Service> </Server> 2. Farm deployment modification shown below: <!-- master node --> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"> <Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer" tempDir="/home/myhome/Apache/bhosts/tc-06/deimos-host/temp-dir/" deployDir="/home/myhome/Apache/bhosts/tc-06/deimos-host/webapps/" watchDir="/home/myhome/Apache/bhosts/tc-06/deimos-host/watch-dir/" watchEnabled="true"/> </Cluster> <!-- slave nodes --> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"> <Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer" tempDir="/home/myhome/Apache/bhosts/tc-06/mars-host/temp-dir/" deployDir="/home/myhome/Apache/bhosts/tc-06/mars-host/webapps/" watchDir="/home/myhome/Apache/bhosts/tc-06/mars-host/watch-dir/" watchEnabled="false"/> </Cluster> 3. startup.sh, shutdown.sh, and setenv.sh files for Tomcat. The directory for $CATALINA_BASE is changed per node. The JMX port is changed per node. #!/bin/bash export CATALINA_BASE=/home/myhome/Apache/bhosts/tc-06/deimos-host export CATALINA_HOME=/home/myhome/Apache/apache-tomcat-6.0.29 $CATALINA_HOME/bin/startup.sh #!/bin/bash export CATALINA_BASE=/home/myhome/Apache/bhosts/tc-06/deimos-host export CATALINA_HOME=/home/myhome/Apache/apache-tomcat-6.0.29 $CATALINA_HOME/bin/shutdown.sh #!/bin/bash export CATALINA_OPTS="-Djava.library.path=/home/myhome/Apache/apache-tomcat-6.0.29/bin/libs -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9004 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false" export JAVA_OPTS="-Dlog4j.home=$CATALINA_BASE/logs" 4. web.xml and context.xml for the test application <?xml version="1.0" encoding="UTF-8"?> <web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation= "http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"> <distributable/> <context-param> <description>Properties file for the PetFactory</description> <param-name>properties</param-name> <param-value>pets.properties</param-value> </context-param> <listener> <description>ServletContextListener</description> <listener-class>rpets.utils.PetFactoryListener</listener-class> </listener> <servlet> <servlet-name>Personalize</servlet-name> <servlet-class>rpets.controller.Personalize</servlet-class> </servlet> <servlet> <servlet-name>Leave</servlet-name> <servlet-class>rpets.controller.Leave</servlet-class> </servlet> <servlet> <servlet-name>Randomize</servlet-name> <servlet-class>rpets.controller.Randomize</servlet-class> </servlet> <servlet> <servlet-name>Finalize</servlet-name> <servlet-class>rpets.controller.Finalize</servlet-class> </servlet> <servlet-mapping> <servlet-name>Personalize</servlet-name> <url-pattern>/protected/Personalize</url-pattern> </servlet-mapping> <servlet-mapping> <servlet-name>Leave</servlet-name> <url-pattern>/Leave</url-pattern> </servlet-mapping> <servlet-mapping> <servlet-name>Randomize</servlet-name> <url-pattern>/protected/Randomize</url-pattern> </servlet-mapping> <servlet-mapping> <servlet-name>Finalize</servlet-name> <url-pattern>/protected/Finalize</url-pattern> </servlet-mapping> <session-config> <session-timeout> 30 </session-timeout> </session-config> <welcome-file-list> <welcome-file>index.jsp</welcome-file> </welcome-file-list> <security-constraint> <display-name>User</display-name> <web-resource-collection> <web-resource-name>RandomPets</web-resource-name> <description>generate a random pet</description> <url-pattern>/protected/*</url-pattern> </web-resource-collection> <auth-constraint> <description>basic user of the web site</description> <role-name>user</role-name> </auth-constraint> </security-constraint> <login-config> <auth-method>FORM</auth-method> <realm-name>Random Pets</realm-name> <form-login-config> <form-login-page>/login.html</form-login-page> <form-error-page>/errorlogin.html</form-error-page> </form-login-config> </login-config> <security-role> <description>basic user of web site</description> <role-name>user</role-name> </security-role> </web-app> <?xml version="1.0" encoding="UTF-8"?> <Context antiJARLocking="true" path="/RPets"> <Resource name="jdbc/auth" description="Pet authentication" type="javax.sql.DataSource" auth="Container" driverClassName="com.mysql.jdbc.Driver" maxActive="10" maxIdle="3" maxWait="10000" password="******" url="jdbc:mysql://localhost/petauth" validationQuery="SELECT 1" username="******"/> <Realm className="org.apache.catalina.realm.DataSourceRealm" userTable="users" userNameCol="username" userCredCol="password" userRoleTable="roles" roleNameCol="rolename" localDataSource="true" dataSourceName="jdbc/auth"/> </Context> 5. Apache configuration files: # workers.properties worker.list=jk-status,jk-manager,lb,deimos,mars,phobos worker.jk-status.type=status worker.jk-status.read_only=true worker.jk-manager.type=status worker.template.type=ajp13 worker.template.host=[my-ip-address] worker.template.socket_connect_timeout=5000 worker.template.socket_keepalive=true worker.template.ping_mode=A worker.template.ping_timeout=10000 worker.template.connection_pool_minsize=0 worker.template.connection_pool_timeout=600 worker.template.reply_timeout=300000 worker.template.recovery_options=3 worker.deimos.reference=worker.template worker.deimos.port=8019 worker.mars.reference=worker.template worker.mars.port=8029 worker.phobos.reference=worker.template worker.phobos.port=8039 worker.lb.type=lb worker.lb.error_escalation_time=0 worker.lb.max_reply_timeouts=10 worker.lb.balance_workers=deimos,mars,phobos # Local uriworkermap.properties file /examples=lb /examples/*=lb /docs=lb /docs/*=lb /RPets=lb /RPets/*=lb # This will be included in the main httpd.conf file LoadModule jk_module modules/mod_jk.so <IfModule jk_module> JkWorkersFile conf.d/workers.properties JkLogFile logs/mod_jk.log JkLogLevel info JkShmFile /var/run/httpd/mod_jk.shm JkOptions +RejectUnsafeURI JkWatchdogInterval 60 <Location /jk-status> JkMount jk-status Order deny,allow Deny from all Allow from 127.0.0.1 </Location> <Location /jk-manager> JkMount jk-manager Order deny,allow Deny from all Allow from 127.0.0.1 </Location> JkMountFile conf.d/uriworkermap.properties </IfModule> --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org