Hi, This turned out to be due to enabling log rotation in the agent (parameter: catalina_rotate_log: 'YES'). We switched to using Log4J logging in Tomcat rather than JULI as it suits our environment better. We were able to use the DailyRollingFileAppender - so the manual rotation of catalina.out by the agent was no longer necessary. After turning this off the problem went away.
I hope this helps someone in the future. Thanks for all the help. Best Regards, Brett -----Original Message----- From: Brett Delle Grazie Sent: Fri 09/07/2010 16:01 To: General Linux-HA mailing list Subject: RE: [Linux-HA] Tomcat Resource Agent always leaves dead process on stop or restart Hi, Yes I meant parent of the process is init. Timeout for start operation is: 120 seconds - process is still around (so is tomcat) after this. heartbeat-libs - taken from linbit repo packages where we have a support contract as we use DRBD in other stuff - I didn't realise they were renamed to cluster-libs on clusterlabs.org. Hmm. I can update but can the packages on clusterlabs still use heartbeat or will I need to switch to corosync? The log from pacemaker / heartbeat is: /var/log/ha-debug Jul 09 15:39:47 fmp-dun-tapp1 lrmd: [4264]: info: rsc:tomcat_tc1:0:184: start Jul 09 15:39:47 fmp-dun-tapp1 attrd: [4266]: info: attrd_ha_callback: flush message from fmp-dun-tapp2 Jul 09 15:39:47 fmp-dun-tapp1 attrd: [4266]: info: attrd_ha_callback: flush message from fmp-dun-tapp2 Jul 09 15:39:47 fmp-dun-tapp1 attrd: [4266]: info: attrd_ha_callback: flush message from fmp-dun-tapp2 Jul 09 15:39:47 fmp-dun-tapp1 attrd: [4266]: info: attrd_ha_callback: flush message from fmp-dun-tapp2 Jul 09 15:39:47 fmp-dun-tapp1 attrd: [4266]: info: attrd_ha_callback: flush message from fmp-dun-tapp2 Jul 09 15:39:47 fmp-dun-tapp1 attrd: [4266]: info: attrd_ha_callback: flush message from fmp-dun-tapp2 Jul 09 15:39:47 fmp-dun-tapp1 attrd: [4266]: info: attrd_ha_callback: flush message from fmp-dun-tapp2 Jul 09 15:39:47 fmp-dun-tapp1 attrd: [4266]: info: attrd_ha_callback: flush message from fmp-dun-tapp2 Jul 09 15:39:48 fmp-dun-tapp1 attrd: [4266]: info: attrd_ha_callback: flush message from fmp-dun-tapp2 Jul 09 15:39:48 fmp-dun-tapp1 attrd: [4266]: info: attrd_ha_callback: flush message from fmp-dun-tapp2 Jul 09 15:39:48 fmp-dun-tapp1 attrd: [4266]: info: attrd_ha_callback: flush message from fmp-dun-tapp2 Jul 09 15:39:51 fmp-dun-tapp1 crmd: [4267]: info: process_lrm_event: LRM operation tomcat_tc1:0_start_0 (call=184, rc=0, cib-update=208, confirmed=true) ok Jul 09 15:39:51 fmp-dun-tapp1 crmd: [4267]: info: do_lrm_rsc_op: Performing key=24:225:0:2c2c0209-48a9-40d0-b4f9-53ea1adcd584 op=tomcat_tc1:0_monitor_10000 ) Jul 09 15:39:51 fmp-dun-tapp1 lrmd: [4264]: info: rsc:tomcat_tc1:0:185: monitor Jul 09 15:39:51 fmp-dun-tapp1 crmd: [4267]: info: process_lrm_event: LRM operation tomcat_tc1:0_monitor_10000 (call=185, rc=0, cib-update=209, confirmed=false) ok Jul 09 15:40:16 fmp-dun-tapp1 lrmd: [4264]: info: rsc:cmon_html0:0:174: monitor Jul 09 15:42:12 fmp-dun-tapp1 cib: [4263]: info: cib_stats: Processed 198 operations (909.00us average, 0% utilization) in the last 10min Jul 09 15:47:43 fmp-dun-tapp1 lrmd: [4264]: info: cancel_op: operation monitor[185] on ocf::tomcat::tomcat_tc1:0 for client 4267, its parameters: CRM_meta_interval=[10000] catalina_home=[/opt/tomcat] catalina_base=[/home/tomcat/tc-1] tomcat_user=[tomcat] catalina_pid=[/home/tomcat/tc-1/temp/tomcat.pid] catalina_rotate_log=[YES] CRM_meta_timeout=[30000] CRM_meta_clone_max=[2] crm_feature_set=[3.0.1] java_home=[/usr/lib/jvm/java] CRM_meta_globally_unique=[false] CRM_meta_name=[monitor] script_log=[/home/tomcat/tc-1/logs/tc-1.log] statusurl=[http://127.0.0.1:10305/exam cancelled Jul 09 15:47:43 fmp-dun-tapp1 crmd: [4267]: info: do_lrm_rsc_op: Performing key=24:226:0:2c2c0209-48a9-40d0-b4f9-53ea1adcd584 op=tomcat_tc1:0_stop_0 ) Jul 09 15:47:43 fmp-dun-tapp1 lrmd: [4264]: info: rsc:tomcat_tc1:0:186: stop Jul 09 15:47:43 fmp-dun-tapp1 crmd: [4267]: info: process_lrm_event: LRM operation tomcat_tc1:0_monitor_10000 (call=185, status=1, cib-update=0, confirmed=true) Cancelled Jul 09 15:47:43 fmp-dun-tapp1 cib: [31183]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-11.raw Jul 09 15:47:43 fmp-dun-tapp1 cib: [31183]: info: write_cib_contents: Wrote version 0.259.0 of the CIB to disk (digest: e028d9e440a93208328ecb4eada8fdf6) Jul 09 15:47:43 fmp-dun-tapp1 cib: [31183]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.8JcqTH (digest: /var/lib/heartbeat/crm/cib.V73jBK) Jul 09 15:47:49 fmp-dun-tapp1 crmd: [4267]: info: process_lrm_event: LRM operation tomcat_tc1:0_stop_0 (call=186, rc=0, cib-update=210, confirmed=true) ok Jul 09 15:47:50 fmp-dun-tapp1 crmd: [4267]: info: do_lrm_rsc_op: Performing key=23:227:0:2c2c0209-48a9-40d0-b4f9-53ea1adcd584 op=tomcat_tc1:0_start_0 ) Jul 09 15:47:50 fmp-dun-tapp1 lrmd: [4264]: info: rsc:tomcat_tc1:0:187: start Jul 09 15:47:54 fmp-dun-tapp1 crmd: [4267]: info: process_lrm_event: LRM operation tomcat_tc1:0_start_0 (call=187, rc=0, cib-update=211, confirmed=true) ok Jul 09 15:47:54 fmp-dun-tapp1 crmd: [4267]: info: do_lrm_rsc_op: Performing key=24:227:0:2c2c0209-48a9-40d0-b4f9-53ea1adcd584 op=tomcat_tc1:0_monitor_10000 ) Jul 09 15:47:54 fmp-dun-tapp1 lrmd: [4264]: info: rsc:tomcat_tc1:0:188: monitor Jul 09 15:47:54 fmp-dun-tapp1 crmd: [4267]: info: process_lrm_event: LRM operation tomcat_tc1:0_monitor_10000 (call=188, rc=0, cib-update=212, confirmed=false) ok Process list is: root 23162 1 0 15:39 ? 00:00:00 /bin/sh /usr/lib/ocf/resource.d//intact/tomcat start root 31372 1 0 15:47 ? 00:00:00 /bin/sh /usr/lib/ocf/resource.d//intact/tomcat start tomcat 31408 1 0 15:47 ? 00:00:03 /usr/lib/jvm/java/bin/java -Djava.util.logging.config.file=/home/tomcat/tc-1/conf/logging.properties ... snip .. org.apache.catalina.startup.Bootstrap start There was a 'crm resource restart cl_tomcat_tc1' issued at 15:47. In the above ha-debug log you can clearly see the lrmd starting tomcat at both points in time (15:39 and 15:47) and receiving a successful start ok response. -----Original Message----- From: Dejan Muhamedagic [mailto:deja...@fastmail.fm] Sent: Fri 09/07/2010 14:46 To: General Linux-HA mailing list Subject: Re: [Linux-HA] Tomcat Resource Agent always leaves dead process on stop or restart Hi, On Fri, Jul 09, 2010 at 01:04:09PM +0100, Brett Delle Grazie wrote: > Hi, > > Yes checked both logs: > > Catalina.out specifies normal (successful) Tomcat startup. > > tc-1.log (log from backgrounded start/stop operations): > > Doesn't give anything unusual: > 2010/07/09 09:42:13: start =========================== > 2010/07/09 10:20:46: stop ########################### > 2010/07/09 10:27:35: start =========================== > 2010/07/09 12:50:20: stop ########################### > 2010/07/09 12:50:26: start =========================== > > Yes, I realise these are from later runs but the same thing is still > occurring. > > Is it possible that the start operation doesn't close of one of > the file descriptors and is left 'hanging' - even though > it exits (at least from the perspective of pacemaker)? > > Would this explain the ownership of 'init' by the 'tomcat > start' process instead of by pacemaker? No. lrmd kills the process if it doesn't exit within the timeout. By "ownership" I guess you mean the parent process. The RA process (/usr/lib/ocf/.../tomcat start) is a child of the lrmd. init can become its parent only if lrmd exits. What is the timeout for that start operation set to? Does the process remain even after that timeout? What happens to lrmd? > > > > heartbeat-libs-3.0.3-1 Where does that come from? Normally, you should have cluster-libs. Perhaps you need to update. Thanks, Dejan > Thanks, > > Best Regards, > > Brett > > > -----Original Message----- > From: Dejan Muhamedagic [mailto:deja...@fastmail.fm] > Sent: Fri 09/07/2010 12:54 > To: General Linux-HA mailing list > Subject: Re: [Linux-HA] Tomcat Resource Agent always leaves dead process on > stop or restart > > Hi, > > On Fri, Jul 09, 2010 at 12:29:40PM +0100, Brett Delle Grazie wrote: > > > > Hi, > > > > Now we come to the fun part... > > > > When I first started looking at this I thought the monitor code in the > > agent was wrong: > > > > ############################################################################ > > # Check tomcat process and service availability > > monitor_tomcat() > > { > > isalive_tomcat || > > return $OCF_NOT_RUNNING > > isrunning_tomcat || > > return $OCF_NOT_RUNNING > > return $OCF_SUCCESS > > } > > > > Both pgrep and wget return 0 if successful, thus so do isalive_tomcat and > > isrunning_tomcat. > > However this appears correct. > > > > So I'm _really_ confused about why this is not exiting. > > > > Any ideas? > > The logs should say. Did you check the tomcat logs too? > > Thanks, > > Dejan > > > Thanks, > > > > Regards, > > > > Brett > > > > > > -----Original Message----- > > From: Dejan Muhamedagic [mailto:deja...@fastmail.fm] > > Sent: Fri 09/07/2010 11:53 > > To: General Linux-HA mailing list > > Subject: Re: [Linux-HA] Tomcat Resource Agent always leaves dead process on > > stop or restart > > > > Hi, > > > > On Fri, Jul 09, 2010 at 11:41:37AM +0100, Brett Delle Grazie wrote: > > > Hi Dejan, > > > > > > Thanks for your response. > > > > > > You are correct the backgrounded process used to start tomcat by the > > > resource agent isn't exiting the way it should - the question is why? > > > > > > Ignore the incorrect date on the example - I killed the wrong leftover > > > process before setting up the example. > > > > > > restarting tomcat is performed by: > > > > > > crm resource restart cl_tomcat_tc1 > > > > > > To the best of my knowledge this performs a 'stop' and then a 'start'. > > > > Right. Note that "stop" won't run before the current action on > > the resource is done. > > > > > Where cl_tomcat_tc1 is a clone tomcat resource. > > > > > > Any ideas why the backgrounded process doesn't exit? > > > > The start action is like this: > > > > java ... start & > > while not monitor: > > sleep > > > > If the monitor never succeeds, then lrmd will kill the process > > once the timeout for the start operation expires. At any rate, > > lrmd always makes sure that there's only one operation on the > > resource at the time. > > > > Thanks, > > > > Dejan > > > > > Thanks, > > > > > > Best Regards, > > > > > > Brett > > > > > > > > > > > > -----Original Message----- > > > From: Dejan Muhamedagic [mailto:deja...@fastmail.fm] > > > Sent: Fri 09/07/2010 11:32 > > > To: General Linux-HA mailing list > > > Subject: Re: [Linux-HA] Tomcat Resource Agent always leaves dead process > > > on stop or restart > > > > > > Hi, > > > > > > On Thu, Jul 08, 2010 at 10:35:57AM +0100, Brett Delle Grazie wrote: > > > > > > > > Hi, > > > > > > > > I'm using RHEL5.5 in a Heartbeat/Pacemaker cluster managing Tomcat and > > > > Apache HTTPD on two nodes using the ocf:heartbeat:tomcat resource agent > > > > for Tomcat. > > > > > > > > Specific versions: > > > > resource-agents 1.0.3-1 > > > > heartbeat-libs-3.0.3-1 > > > > heartbeat-3.0.3-1 > > > > pacemaker-1.0.8-1.0hg20100317.8debc1902e13 > > > > Tomcat 6.0.26 (downloaded from source). > > > > > > > > I have modified the Tomcat resource agent to be capable of > > > > controlling multiple Tomcat instances by exporting > > > > CATALINA_BASE as well as CATALINA_HOME - these are the only > > > > > > > > > > changes I've made to the resource agent (this is why the agent > > > > path is 'intact' instead of 'heartbeat' in the process list > > > > below) - diff attached. > > > > > > > > When manually issuing a restart of the clone resource on tomcat > > > > I'm left with a dead 'start' process: > > > > > > Doesn't look dead to me, just that it didn't exit. > > > > > > > (before restart): > > > > [r...@fmp-dun-tapp1 ~]# ps -efH | grep [t]omcat > > > > root 22754 21037 0 10:09 pts/0 00:00:00 grep tomcat > > > > root 5058 1 0 Jul07 ? 00:00:00 /bin/sh > > > > /usr/lib/ocf/resource.d//intact/tomcat start > > > > tomcat 5101 1 0 Jul07 ? 00:00:19 > > > > /usr/lib/jvm/java/bin/java > > > > -Djava.util.logging.config.file=/home/tomcat/tc-1/conf/logging.properties > > > > -Dname=tomcat -Djava.awt.headless=true -Djava.library.path=/usr/lib64 > > > > -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager > > > > -Xmx1024M -Djava.endorsed.dirs=/opt/tomcat/endorsed -classpath > > > > /opt/tomcat/bin/bootstrap.jar -Dcatalina.base=/home/tomcat/tc-1 > > > > -Dcatalina.home=/opt/tomcat -Djava.io.tmpdir=/home/tomcat/tc-1/temp > > > > org.apache.catalina.startup.Bootstrap start > > > > > > > > (after restart): > > > > [r...@fmp-dun-tapp1 ~]# ps -efH | grep [t]omcat > > > > root 5058 1 0 Jul07 ? 00:00:00 /bin/sh > > > > /usr/lib/ocf/resource.d//intact/tomcat start > > > > > > This looks like an old process, judging by the date. Perhaps you > > > killed (using -9) some processes so this one remained hanging? > > > Otherwise, this is not possible, i.e. only one operation on a > > > resource is run. > > > > > > > root 2271 1 0 10:26 ? 00:00:00 /bin/sh > > > > /usr/lib/ocf/resource.d//intact/tomcat start > > > > tomcat 2307 1 21 10:26 ? 00:00:02 > > > > /usr/lib/jvm/java/bin/java > > > > -Djava.util.logging.config.file=/home/tomcat/tc-1/conf/logging.properties > > > > -Dname=tomcat -Djava.awt.headless=true -Djava.library.path=/usr/lib64 > > > > -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager > > > > -Xmx1024M -Djava.endorsed.dirs=/opt/tomcat/endorsed -classpath > > > > /opt/tomcat/bin/bootstrap.jar -Dcatalina.base=/home/tomcat/tc-1 > > > > -Dcatalina.home=/opt/tomcat -Djava.io.tmpdir=/home/tomcat/tc-1/temp > > > > org.apache.catalina.startup.Bootstrap start > > > > > > > > Note the two 'tomcat start' processes above. > > > > Each restart produces successively more copies of the 'tocmat start' > > > > process. > > > > > > What is a "restart"? How does it happen? > > > > > > > Does anyone know why this would occur? I thought the call to > > > > 'catalina.sh start' which is backgrounded in tomcat_start > > > > function in resource should exit after starting Tomcat - but > > > > apparently it doesn't. > > > > > > That's strange. The '&' at the end of the line certainly makes to > > > run in background. Otherwise, the start action goes into infinite > > > loop waiting for the monitor of the resource to succeed. If it > > > never does, then lrmd will timeout and kill the process. > > > > > > Thanks, > > > > > > Dejan > > > > > > > > > > Any help / trouble-shooting tips appreciated. > > > > > > > > Thanks, > > > > > > > > Best Regards, > > > > > > > > Brett > > > > > > > > > > > > > > > > > > > > ______________________________________________________________________ > > > > This email has been scanned by the MessageLabs Email Security System. > > > > For more information please visit http://www.messagelabs.com/email > > > > ______________________________________________________________________ > > > > > > > --- tomcat 2010-07-08 09:34:13.000000000 +0100 > > > > +++ tomcat.intact 2010-07-08 10:24:51.000000000 +0100 > > > > @@ -29,7 +29,9 @@ > > > > # OCF_RESKEY_tomcat_user - A user name to start a resource. Default > > > > is root > > > > # OCF_RESKEY_statusurl - URL for state confirmation. Default is > > > > http://127.0.0.1:8080 > > > > # OCF_RESKEY_java_home - Home directory of the Java. Default is None > > > > +# OCF_RESKEY_java_opts - Options to parse to Java. Always adds > > > > -Dname=OCF_RESKEY_tomcat_name > > > > # OCF_RESKEY_catalina_home - Home directory of Tomcat. Default is > > > > None > > > > +# OCF_RESKEY_catalina_base - Base directory of Tomcat. Default is > > > > None > > > > # OCF_RESKEY_catalina_pid - A PID file name of Tomcat. Default is > > > > OCF_RESKEY_catalina_home/logs/catalina.pid > > > > # OCF_RESKEY_tomcat_start_opts - Start options of the tomcat. > > > > Default is None. > > > > # OCF_RESKEY_catalina_opts - CATALINA_OPTS environment variable. > > > > Default is None. > > > > @@ -147,11 +149,12 @@ > > > > >> "$TOMCAT_CONSOLE" 2>&1 & > > > > else > > > > su - -s /bin/sh "$RESOURCE_TOMCAT_USER" \ > > > > - -c "export JAVA_HOME=${OCF_RESKEY_java_home};\n > > > > - export JAVA_OPTS=-Dname=${TOMCAT_NAME};\n > > > > - export > > > > CATALINA_HOME=${OCF_RESKEY_catalina_home};\n > > > > - export > > > > CATALINA_PID=${OCF_RESKEY_catalina_pid};\n > > > > - export > > > > CATALINA_OPTS=\"${OCF_RESKEY_catalina_opts}\";\n > > > > + -c "export JAVA_HOME=${OCF_RESKEY_java_home};\ > > > > + export JAVA_OPTS=\"-Dname=${TOMCAT_NAME} > > > > ${OCF_RESKEY_java_opts}\";\ > > > > + export > > > > CATALINA_HOME=${OCF_RESKEY_catalina_home};\ > > > > + export > > > > CATALINA_BASE=${OCF_RESKEY_catalina_base};\ > > > > + export > > > > CATALINA_PID=${OCF_RESKEY_catalina_pid};\ > > > > + export > > > > CATALINA_OPTS=\"${OCF_RESKEY_catalina_opts}\";\ > > > > $CATALINA_HOME/bin/catalina.sh start > > > > ${OCF_RESKEY_tomcat_start_opts}" \ > > > > >> "$TOMCAT_CONSOLE" 2>&1 & > > > > fi > > > > @@ -182,10 +185,11 @@ > > > > eval $tomcat_stop_cmd >> "$TOMCAT_CONSOLE" 2>&1 > > > > else > > > > su - -s /bin/sh "$RESOURCE_TOMCAT_USER" \ > > > > - -c "export JAVA_HOME=${OCF_RESKEY_java_home};\n > > > > - export JAVA_OPTS=-Dname=${TOMCAT_NAME};\n > > > > - export > > > > CATALINA_HOME=${OCF_RESKEY_catalina_home};\n > > > > - export > > > > CATALINA_PID=${OCF_RESKEY_catalina_pid};\n > > > > + -c "export JAVA_HOME=${OCF_RESKEY_java_home};\ > > > > + export JAVA_OPTS=\"-Dname=${TOMCAT_NAME} > > > > ${OCF_RESKEY_java_opts}\";\ > > > > + export > > > > CATALINA_HOME=${OCF_RESKEY_catalina_home};\ > > > > + export > > > > CATALINA_BASE=${OCF_RESKEY_catalina_base};\ > > > > + export > > > > CATALINA_PID=${OCF_RESKEY_catalina_pid};\ > > > > $CATALINA_HOME/bin/catalina.sh stop" \ > > > > >> "$TOMCAT_CONSOLE" 2>&1 > > > > fi > > > > @@ -316,6 +320,14 @@ > > > > <content type="string" default="" /> > > > > </parameter> > > > > > > > > +<parameter name="java_opts" unique="0"> > > > > +<longdesc lang="en"> > > > > +Java options > > > > +</longdesc> > > > > +<shortdesc>Java options</shortdesc> > > > > +<content type="string" default="" /> > > > > +</parameter> > > > > + > > > > <parameter name="catalina_home" unique="1" required="1"> > > > > <longdesc lang="en"> > > > > Home directory of Tomcat > > > > @@ -324,6 +336,14 @@ > > > > <content type="string" default="" /> > > > > </parameter> > > > > > > > > +<parameter name="catalina_base" unique="1"> > > > > +<longdesc lang="en"> > > > > +Instance directory of Tomcat > > > > +</longdesc> > > > > +<shortdesc>Instance directory of Tomcat</shortdesc> > > > > +<content type="string" default="" /> > > > > +</parameter> > > > > + > > > > <parameter name="catalina_pid" unique="1"> > > > > <longdesc lang="en"> > > > > A PID file name of Tomcat > > > > @@ -397,9 +417,10 @@ > > > > RESOURCE_STATUSURL="${OCF_RESKEY_statusurl-http://127.0.0.1:8080}"; > > > > > > > > JAVA_HOME="${OCF_RESKEY_java_home}" > > > > -JAVA_OPTS="-Dname=$TOMCAT_NAME" > > > > +JAVA_OPTS="-Dname=$TOMCAT_NAME ${OCF_RESKEY_java_opts}" > > > > SEARCH_STR="\\""${JAVA_OPTS}" > > > > CATALINA_HOME="${OCF_RESKEY_catalina_home}" > > > > +CATALINA_BASE="${OCF_RESKEY_catalina_base}" > > > > > > > > CATALINA_PID="${OCF_RESKEY_catalina_pid-$CATALINA_HOME/logs/catalina.pid}" > > > > > > > > TOMCAT_START_OPTS="${OCF_RESKEY_tomcat_start_opts}" > > > > @@ -407,7 +428,7 @@ > > > > CATALINA_ROTATE_LOG="${OCF_RESKEY_catalina_rotate_log-NO}" > > > > CATALINA_ROTATETIME="${OCF_RESKEY_catalina_rotatetime-86400}" > > > > > > > > -export JAVA_HOME JAVA_OPTS CATALINA_HOME CATALINA_PID CATALINA_OPTS > > > > +export JAVA_HOME JAVA_OPTS CATALINA_HOME CATALINA_BASE CATALINA_PID > > > > CATALINA_OPTS > > > > > > > > JAVA=${JAVA_HOME}/bin/java > > > > > > > > > > > _______________________________________________ > > > > Linux-HA mailing list > > > > Linux-HA@lists.linux-ha.org > > > > http://lists.linux-ha.org/mailman/listinfo/linux-ha > > > > See also: http://linux-ha.org/ReportingProblems > > > > > > > > > > > > > > > > > > ______________________________________________________________________ > > > This email has been scanned by the MessageLabs Email Security System. > > > For more information please visit http://www.messagelabs.com/email > > > ______________________________________________________________________ > > > > > > > _______________________________________________ > > > Linux-HA mailing list > > > Linux-HA@lists.linux-ha.org > > > http://lists.linux-ha.org/mailman/listinfo/linux-ha > > > See also: http://linux-ha.org/ReportingProblems > > > > > > > > > > > > ______________________________________________________________________ > > This email has been scanned by the MessageLabs Email Security System. > > For more information please visit http://www.messagelabs.com/email > > ______________________________________________________________________ > > _______________________________________________ > > Linux-HA mailing list > > Linux-HA@lists.linux-ha.org > > http://lists.linux-ha.org/mailman/listinfo/linux-ha > > See also: http://linux-ha.org/ReportingProblems > > > > ______________________________________________________________________ > This email has been scanned by the MessageLabs Email Security System. > For more information please visit http://www.messagelabs.com/email > ______________________________________________________________________ > _______________________________________________ > Linux-HA mailing list > Linux-HA@lists.linux-ha.org > http://lists.linux-ha.org/mailman/listinfo/linux-ha > See also: http://linux-ha.org/ReportingProblems ______________________________________________________________________ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email ______________________________________________________________________ _______________________________________________ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
