Hi, issue still not fixed. Tomcat session replication is not able to
replicate the key from node to node, please find below the error,
taking into consideration that there is no network disconnection
between 2 nodes
07-Feb-2017 16:36:06.186 SEVERE [http-nio-8080-exec-8]
org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryInfo
Unable to replicate backup
key:58291D242C742A8A4B1657BA42C831A4.TomcatNode2 to
backup:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
114, 43, 102}:4000,{10, 114, 43, 102},4000, alive=68841350,
securePort=-1, UDP Port=-1, id={85 5 -62 -66 106 -12 64 12 -102 -14
-85 -87 15 9 -51 -112 }, payload={}, command={}, domain={}, ].
Reason:Operation has timed out(3000 ms.).; Faulty members:tcp://{10,
114, 43, 102}:4000;
org.apache.catalina.tribes.ChannelException: Operation has timed
out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 102}:4000;
at
org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108)
at
org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
at
org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54)
at
org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
at
org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
at
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233)
at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186)
at
org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryInfo(LazyReplicatedMap.java:170)
at
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(AbstractReplicatedMap.java:1040)
at
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(AbstractReplicatedMap.java:1024)
at org.apache.catalina.session.ManagerBase.add(ManagerBase.java:647)
at org.apache.catalina.session.StandardSession.setId(StandardSession.java:374)
at org.apache.catalina.ha.session.DeltaSession.setId(DeltaSession.java:279)
at org.apache.catalina.session.ManagerBase.createSession(ManagerBase.java:708)
at org.apache.catalina.connector.Request.doGetSession(Request.java:2936)
at org.apache.catalina.connector.Request.getSession(Request.java:2260)
at
org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:895)
at
javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:231)
at
org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:568)
at
org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:513)
at
org.apache.jasper.runtime.PageContextImpl.initialize(PageContextImpl.java:137)
at
org.apache.jasper.runtime.JspFactoryImpl.internalGetPageContext(JspFactoryImpl.java:109)
at
org.apache.jasper.runtime.JspFactoryImpl.getPageContext(JspFactoryImpl.java:60)
at org.apache.jsp.WEB_002dINF.jsp._401_jsp._jspService(_401_jsp.java:100)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:438)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:396)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:340)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:719)
at
org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:467)
at
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:390)
at
org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:317)
at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:445)
at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:304)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:181)
at
org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:194)
at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:318)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at
org.apache.catalina.valves.StuckThreadDetectionValve.invoke(StuckThreadDetectionValve.java:208)
at
org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:521)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1096)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:674)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1456)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Unknown Source)
On Tue, Feb 7, 2017 at 1:02 PM, Kreuser, Peter <pkreu...@airplus.com> wrote:
> Fady,
>
> Sorry for top posting.
>
> If I remember correctly, the Cluster Element goes into the Container and not
> the Host.
> Plus I see in our (working) case, a DeltaManager and a
> JvmRouteSessionIDBinderListener
>
> <Manager className="org.apache.catalina.ha.session.DeltaManager"
> expireSessionsOnShutdown="false"
> notifyListenersOnReplication="true"/>
> ...
> <ClusterListener
> className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
>
> Besides this, only ports, limits and values are different.
>
> You may want to filter out the replication for static resources as gifs jpg
> or css.
> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
> filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.css;.*\.txt;" />
>
> Best regards
>
> Peter
>
>> below is the server.xml configuration, as mentioened earlier the issue
>> is related to the cluster configuration, and as per my research i can
>> see that some users are facing the same issue but i didnt found the
>> solution of it
>>
>>
>> <?xml version='1.0' encoding='utf-8'?>
>> <!--
>> Licensed to the Apache Software Foundation (ASF) under one or more
>> contributor license agreements. See the NOTICE file distributed with
>> this work for additional information regarding copyright ownership.
>> The ASF licenses this file to You under the Apache License, Version 2.0
>> (the "License"); you may not use this file except in compliance with
>> the License. You may obtain a copy of the License at
>>
>> http://www.apache.org/licenses/LICENSE-2.0
>>
>> Unless required by applicable law or agreed to in writing, software
>> distributed under the License is distributed on an "AS IS" BASIS,
>> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>> See the License for the specific language governing permissions and
>> limitations under the License.
>> -->
>> <!-- Note: A "Server" is not itself a "Container", so you may not
>> define subcomponents such as "Valves" at this level.
>> Documentation at /docs/config/server.html
>> -->
>> <Server port="8005" shutdown="SHUTDOWN">
>> <Listener className="org.apache.catalina.startup.VersionLoggerListener" />
>> <!-- Security listener. Documentation at /docs/config/listeners.html
>> <Listener className="org.apache.catalina.security.SecurityListener" />
>> -->
>> <!--APR library loader. Documentation at /docs/apr.html -->
>> <Listener className="org.apache.catalina.core.AprLifecycleListener"
>> SSLEngine="on" />
>> <!-- Prevent memory leaks due to use of particular java/javax APIs-->
>> <Listener
>> className="org.apache.catalina.core.JreMemoryLeakPreventionListener"
>> />
>> <Listener
>> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
>> />
>> <Listener
>> className="org.apache.catalina.core.ThreadLocalLeakPreventionListener"
>> />
>>
>> <!-- Global JNDI resources
>> Documentation at /docs/jndi-resources-howto.html
>> -->
>> <GlobalNamingResources>
>> <!-- Editable user database that can also be used by
>> UserDatabaseRealm to authenticate users
>> -->
>> <Resource name="UserDatabase" auth="Container"
>> type="org.apache.catalina.UserDatabase"
>> description="User database that can be updated and saved"
>> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
>> pathname="conf/tomcat-users.xml" />
>> </GlobalNamingResources>
>>
>> <!-- A "Service" is a collection of one or more "Connectors" that share
>> a single "Container" Note: A "Service" is not itself a "Container",
>> so you may not define subcomponents such as "Valves" at this level.
>> Documentation at /docs/config/service.html
>> -->
>> <Service name="Catalina">
>>
>> <!--The connectors can use a shared executor, you can define one
>> or more named thread pools-->
>> <!--
>> <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
>> maxThreads="150" minSpareThreads="4"/>
>> -->
>>
>>
>> <!-- A "Connector" represents an endpoint by which requests are received
>> and responses are returned. Documentation at :
>> Java HTTP Connector: /docs/config/http.html (blocking &
>> non-blocking)
>> Java AJP Connector: /docs/config/ajp.html
>> APR (HTTP/AJP) Connector: /docs/apr.html
>> Define a non-SSL/TLS HTTP/1.1 Connector on port 8080
>> -->
>> <Connector port="8080" protocol="HTTP/1.1"
>> connectionTimeout="60000" maxThreads="500"
>> minSpareThreads="25" maxSpareThreads="75" enableLookups="false"
>> disableUploadTimeout="true" acceptCount="100" redirectPort="8443" />
>> <!-- A "Connector" using the shared thread pool-->
>> <!--
>> <Connector executor="tomcatThreadPool"
>> port="8080" protocol="HTTP/1.1"
>> connectionTimeout="60000"
>> redirectPort="8443" />
>> -->
>> <!-- Define a SSL/TLS HTTP/1.1 Connector on port 8443
>> This connector uses the NIO implementation that requires the JSSE
>> style configuration. When using the APR/native implementation, the
>> OpenSSL style configuration is required as described in the
>> APR/native
>> documentation -->
>> <!--
>> <Connector port="8443"
>> protocol="org.apache.coyote.http11.Http11NioProtocol"
>> maxThreads="150" SSLEnabled="true" scheme="https"
>> secure="true"
>> clientAuth="false" sslProtocol="TLS" />
>> -->
>>
>> <!-- Define an AJP 1.3 Connector on port 8009 -->
>> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
>>
>>
>> <!-- An Engine represents the entry point (within Catalina) that
>> processes
>> every request. The Engine implementation for Tomcat stand alone
>> analyzes the HTTP headers included with the request, and passes them
>> on to the appropriate Host (virtual host).
>> Documentation at /docs/config/engine.html -->
>>
>> <!-- You should set jvmRoute to support load-balancing via AJP ie :
>>
>> -->
>> <Engine name="Catalina" defaultHost="localhost" jvmRoute="TomcatNode1">
>> <!-- Use the LockOutRealm to prevent attempts to guess user passwords
>> via a brute-force attack -->
>> <Realm className="org.apache.catalina.realm.LockOutRealm">
>> <!-- This Realm uses the UserDatabase configured in the global JNDI
>> resources under the key "UserDatabase". Any edits
>> that are performed against this UserDatabase are immediately
>> available for use by the Realm. -->
>> <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
>> resourceName="UserDatabase"/>
>> </Realm>
>>
>> <Host name="localhost" appBase="webapps"
>> unpackWARs="true" autoDeploy="true" startStopThreads="0">
>> <!--For clustering, please take a look at documentation at:
>> /docs/cluster-howto.html (simple how to)
>> /docs/config/cluster.html (reference documentation) -->
>> <!--
>> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
>> -->
>>
>> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
>> channelSendOptions="4">
>> <Channel className="org.apache.catalina.tribes.group.GroupChannel">
>> <Membership
>> className="org.apache.catalina.tribes.membership.McastService"
>> address="228.0.0.4"
>> port="45564"
>> frequency="500"
>> dropTime="9000"/>
>> <Receiver
>> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
>> address="auto"
>> port="4000"
>> autoBind="100"
>> selectorTimeout="5000"
>> maxThreads="6"/>
>>
>> <Sender
>> className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
>> <Transport
>> className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
>> </Sender>
>> <Interceptor
>> className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
>> <Interceptor
>> className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
>> </Channel>
>>
>> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
>> filter=""/>
>> <Valve
>> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
>>
>> <Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer"
>> tempDir="D:/imaljava/TomcatNode1/tmp/war-temp/"
>> deployDir="D:/imaljava/TomcatNode1/tmp/war-deploy/"
>> watchDir="D:/imaljava/TomcatNode1/tmp/war-listen/"
>> watchEnabled="false"/>
>>
>> <ClusterListener
>> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
>> </Cluster>
>> <!-- SingleSignOn valve, share authentication between web
>> applications
>> Documentation at: /docs/config/valve.html -->
>> <!--
>> <Valve className="org.apache.catalina.authenticator.SingleSignOn" />
>> -->
>>
>> <!-- Access log processes all example.
>> Documentation at: /docs/config/valve.html
>> Note: The pattern used is equivalent to using pattern="common"
>> -->
>> <Valve className="org.apache.catalina.valves.AccessLogValve"
>> directory="logs"
>> prefix="localhost_access_log" suffix=".txt"
>> pattern="%h %l %u %t "%r" %s %b" />
>> <Valve className="org.apache.catalina.valves.StuckThreadDetectionValve"
>> threshold="900" />
>>
>> </Host>
>> </Engine>
>> </Service>
>> </Server>
>>
>> On Mon, Feb 6, 2017 at 6:51 PM, André Warnier (tomcat) <a...@ice-sa.com>
>> wrote:
>> > On 06.02.2017 17:45, Fady Haikal wrote:
>> >>
>> >> Hi,
>> >> What is the host OS ? Windows Server 2012
>> >> What is the Tomcat version ? Apache Tomcat/8.0.30
>> >>
>> >> Is this problem new ? was this working before ? how long ? Since
>> >> cluster implementation
>> >>
>> >
>> > I still don't know tribes, but then my non-educated guess at this point
>> > would be that there is something wrong in your configuration.
>> > Can you copy/paste it here ? (remove sensible things like passwords, public
>> > IP addresses etc..)(but not to the point of making it uncheckable).
>> >
>> > Then maybe some tribes-specialist can take over ?
>> >
>> >
>> >>
>> >> Is there actually something listening on that address/port ? Tomcat
>> >> cluster
>> >>
>> >> the Port 4000 is listening and there is no disconnection between 2
>> >> nodes ping and telnet are OK
>> >>
>> >> On Mon, Feb 6, 2017 at 6:42 PM, André Warnier (tomcat) <a...@ice-sa.com>
>> >> wrote:
>> >>>
>> >>> On 06.02.2017 17:24, Fady Haikal wrote:
>> >>>>
>> >>>>
>> >>>> Plz can i get some help here?
>> >>>> This issue is still occurring and it's filling the log file in the
>> >>>> Production server
>> >>>>
>> >>>> Regards,
>> >>>> Fady
>> >>>
>> >>>
>> >>>
>> >>> Hi.
>> >>> If you want quick answers, you should provide more information.
>> >>> What is the host OS ?
>> >>> What is the Tomcat version ?
>> >>> Is this problem new ? was this working before ? how long ?
>> >>>
>> >>> I do not know tribes at all, but according to the logfile below, it seems
>> >>> that something is trying to "ping" the address 10.114.43.103, port 4000,
>> >>> and
>> >>> never getting a response (or at least not within 3000ms).
>> >>> Is there actually something listening on that address/port ?
>> >>> The "netstat" command (available both on Linux and Windows) can tell you.
>> >>> If there is something listening there, can it respond to whatever is
>> >>> ping-ing it ?
>> >>> (routing, firewall, ..)
>> >>>
>> >>>>
>> >>>> On Mon, Feb 6, 2017 at 8:52 AM, Fady Haikal <fadyhai...@gmail.com>
>> >>>> wrote:
>> >>>>>
>> >>>>>
>> >>>>> Guys, we are facing the below errors in Tomcat cluster, please advise
>> >>>>>
>> >>>>>
>> >>>>> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
>> >>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
>> >>>>> Unable to send AbstractReplicatedMap.ping message
>> >>>>> org.apache.catalina.tribes.ChannelException: Operation has timed
>> >>>>> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000;
>> >>>>> at
>> >>>>>
>> >>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108)
>> >>>>> at
>> >>>>>
>> >>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
>> >>>>> at
>> >>>>>
>> >>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54)
>> >>>>> at
>> >>>>>
>> >>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82)
>> >>>>> at
>> >>>>>
>> >>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>> >>>>> at
>> >>>>>
>> >>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
>> >>>>> at
>> >>>>>
>> >>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>> >>>>> at
>> >>>>>
>> >>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
>> >>>>> at
>> >>>>>
>> >>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>> >>>>> at
>> >>>>>
>> >>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233)
>> >>>>> at
>> >>>>>
>> >>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186)
>> >>>>> at org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:99)
>> >>>>> at
>> >>>>>
>> >>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:267)
>> >>>>> at
>> >>>>>
>> >>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:885)
>> >>>>> at
>> >>>>>
>> >>>>> org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:161)
>> >>>>> at
>> >>>>>
>> >>>>> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:697)
>> >>>>>
>> >>>>>
>> >>>>> 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
>> >>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen Unable to
>> >>>>> process request in NioReceiver
>> >>>>> java.io.IOException: A non-blocking socket operation could not be
>> >>>>> completed immediately
>> >>>>> at sun.nio.ch.SocketDispatcher.close0(Native Method)
>> >>>>> at sun.nio.ch.SocketDispatcher.close(Unknown Source)
>> >>>>> at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
>> >>>>> at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source)
>> >>>>> at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source)
>> >>>>> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
>> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
>> >>>>> at sun.nio.ch.SelectorImpl.select(Unknown Source)
>> >>>>> at
>> >>>>>
>> >>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen(NioReceiver.java:272)
>> >>>>> at
>> >>>>>
>> >>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.run(NioReceiver.java:425)
>> >>>>> at java.lang.Thread.run(Unknown Source)
>> >>>>>
>> >>>>> Regards,
>> >>>>> Fady
>> >>>>
>> >>>>
>> >>>>
>> >>>> ---------------------------------------------------------------------
>> >>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> >>>> For additional commands, e-mail: users-h...@tomcat.apache.org
>> >>>>
>> >>>
>> >>>
>> >>> ---------------------------------------------------------------------
>> >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> >>> For additional commands, e-mail: users-h...@tomcat.apache.org
>> >>>
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> >> For additional commands, e-mail: users-h...@tomcat.apache.org
>> >>
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> > For additional commands, e-mail: users-h...@tomcat.apache.org
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
