Hi, André My case is same, my tomcat also has load balancer in front and it has health check to tomcat
-----Original Message----- From: André Warnier (tomcat) [mailto:a...@ice-sa.com] Sent: Wednesday, January 18, 2017 9:41 AM To: users@tomcat.apache.org Subject: Re: FW: tomcat 8080 thread not reduced Hi. I believe that what Philippe mentions below is somewhat different : in his configuration, there is apparently a front-end httpd server, which communicates with Tomcat via AJP and the tomcat AJP Connector. In such a case, the "mod_jk" or "mod_proxy_ajp" connector module inside of httpd, creates a series of connections to the back-end tomcat, and keeps these connections alive in a pool of connections, according to rules which are different, than what applies to your case (which consists of direct HTTP connections from browsers, to the tomcat HTTP connector). So Philippe's case introduces probably an additional level of complexity into the equation. semi-graphically : smith case : client <- HTTP -> tomcat HTTP connector - tomcat thread Philippe's case : client <- HTTP -> apache httpd + ajp module <- AJP -> tomcat AJP connector - tomcat thread The real question is : The tomcat 8.0 HTTP Connector documentation (http://tomcat.apache.org/tomcat-8.0-doc/config/http.html) mentions a series of attributes/parameters (maxThreads, minSpareThreads, connectionTimeout, keepAliveTimeout,..) which, together, would lead one to believe that tomcat is managing the number of threads dynamically (increasing the number of threads (up to maxThreads) when needed to process simultaneous requests, and decreasing the number of threads again (down to minSpareThreads) when less requests are being processed). According to observations however (and to Chris' comments), that does not seem to be the case : once the number of threads has grown to a certain number, it never goes down back down again. It has to be said that the above interpretation was mine, and the current documentation of the HTTP Connector never says explicitly that, on its own, it manages the number of threads dynamically. But it *does* mention a minSpareThreads parameter. On the other hand, when you use an Executor (http://tomcat.apache.org/tomcat-8.0-doc/config/executor.html), then the documentation states explicitly that the number of threads *is* dynamically managed, up and down. I found something else which seems to explain the riddle : http://tomcat.apache.org/migration-6.html#Connector_thread_pools That section explicitly says that since tomcat 6.0, for a Connector without an Executor, the number of threads always grows, and never decreases. And it also explicitly says that the Connector's "minSpareThreads" attribute will be ignored. So in fact the only thing wrong, is the online documentation for the Connectors : the minSpareThreads attribute should be removed (since it is anyway ignored). That seems to have been an oversight ever since tomcat 6.0. As far as I am concerned thus, the mystery is solved. One question which is a bit left open is : What - if any - is the real advantage/disadvantage of having perhaps maxThreads idle threads, as opposed to using an Executor to manage the number of threads dynamically ? But that is probably the kind of question to which the appropriate answer is : "it depends.." On 18.01.2017 07:33, smith wrote: > Thanks, Philippe > > But we never monitored tomcat busy thread count high (we monitored one > minutes interval through nagios to get tomcat manager result, not high than > 10). This is strange > > -----Original Message----- > From: Philippe Busque [mailto:pbus...@mediagrif.com] > Sent: Monday, January 16, 2017 8:09 PM > To: users@tomcat.apache.org > Subject: Re: Re: FW: tomcat 8080 thread not reduced > > We're having a similar issues with our numberous Tomcat instances. > > Our connector config look like this. > > <Connector port="8910" protocol="AJP/1.3" maxThreads="200" > enableLookups="false" redirectPort="8443" acceptCount="100" > connectionTimeout="20000" URIEncoding="UTF-8" /> > > > Sometime, the number of active connection would jump very high (up to 190), > due to some external issues (database lock, etc) and threads would accumulate. > > Even though a connectionTimeout is set, and therefor set keepAliveTimeout as > the same value, threads are never released once the problem is resolved > until Tomcat is restarted. We would end up with maybe 5-10 busy workers, but > 190 idle workers/ threads > > I think the issue is related to how the StandardThreadExecutor is > implemented. The StandardThreadExecutor is a front for the default Java > ThreadPoolExecutor. If I'm not mistaken, ThreadPoolExecutor is distributing > work in round robin fashion among all defined workers, rather than sticking > to the core threads. > > > As a result, should a website has any constant traffic (Apache AJP ping, load > balancer monitoring, normal traffic, etc), all thread will be hit at least > once within the configured keepAliveTimeout, reseting it. So unless the > keepAliveTimeout is set to a very low value, which defeat the purpose, thread > will never be released . > > > This is what I've come to suspect from looking at the StandardThreadExecutor, > but never really had the opportunity to do deeper test with load. > But from Tomcat 6 to tomcat 8, we were never able to decrease the number of > 'idle' workers back from the highest value it had reached. > > > Le 2017-01-16 à 05:24, André Warnier (tomcat) a écrit : >> On 16.01.2017 11:10, smith wrote: >>> We has same problem on dev env that no any traffic to the serive, >> >> Ah. That is /new/ information, which may change the suggestions below. >> It looks like you should really find out what these threads are doing, >> probably by doing a few thread dumps. >> See here e.g. : >> http://stackoverflow.com/questions/18573411/tomcat-thread-dump >> >> Again : we do not know your application, so we can only make guesses >> based on the information that you provide. >> >> will try on dev first >>> >>> -----Original Message----- >>> From: André Warnier (tomcat) [mailto:a...@ice-sa.com] >>> Sent: Monday, January 16, 2017 10:08 AM >>> To: users@tomcat.apache.org >>> Subject: Re: FW: tomcat 8080 thread not reduced >>> >>> On 16.01.2017 09:50, smith wrote: >>>> Busy one is process customer request, do not know what non-busy one >>>> is doing, always keep 120 for many days. I don't think 20s timeout >>>> will not cause so long connection >>>> >>>> -smith >>> >>> And did you actually try it ? >>> >>> We do not know your website or your application, so we cannot tell >>> how many clients there are, what these clients are really requesting, >>> how many requests each client is sending before going away, etc. >>> >>> KeepAlive means that when a client has sent its /last/ request and >>> received the response, one thread is going to remain "not free" (but >>> doing nothing) for the duration of the KeepAlive timeout. This thread >>> will keep waiting, for KeepAliveTimeout seconds, just in case the >>> client would still send another request (which it may never do, >>> depending on the application). >>> >>> Imagine that your application is so that the average client >>> - connects to your site >>> - sends a single HTTP request, which gets processed in 0.1 s >>> - receives the response >>> - and then goes away >>> and that the above sequence happens once every second, from different >>> clients. >>> After one second, there will be one thread waiting for another 19 >>> seconds before becoming free (and potentially destroyed or re-used). >>> After 2 seconds, there will be 2 such threads. After 3 seconds, 3 >>> threads. And so on. After 20 seconds, the first thread will be freed, >>> but there will be 19 other threads still waiting, and one new thread >>> just created. >>> If everything stays perfectly regular like that, your will have >>> /permanently/ 20 threads in existence, even if the minimum is 10. >>> If you change the above so that there is a new client every 0.5 s, >>> you will have permanently 40 threads (of which only 2 maximum are >>> really doing something). >>> >>> The point is : KeepAlive is not "bad", and in some cases having a >>> relatively long KeepAliveTimeout is the right thing to do. Also, >>> having a high number of threads sitting idle is not necessarily a >>> problem. >>> Your own scenario is probably not like the above perfectly regular >>> and irrealistic one above. But there may be a perfectly logical >>> reason why you have so many threads on average, and I am just trying >>> to give you ideas for finding out the reason. >>> >>> >>>> >>>> -----Original Message----- >>>> From: André Warnier (tomcat) [mailto:a...@ice-sa.com] >>>> Sent: Monday, January 16, 2017 8:33 AM >>>> To: users@tomcat.apache.org >>>> Subject: Re: FW: tomcat 8080 thread not reduced >>>> >>>> On 16.01.2017 03:41, Smith Hua wrote: >>>>> >>>>> actually there is not much busy threads, less tahn 10,so i think >>>>> this parameter may has nothing to do with this >>>> >>>> It depends on what you call "busy". What are the busy ones doing ? >>>> and what are the non-busy ones doing ? >>>> >>>> >>>>> -- >>>>> 从myMail的Android专用app所发送 星期一, 16 一月 2017, 03:01上午 +08:00 >>>>> 发件人 André Warnier (tomcat) a...@ice-sa.com : >>>>> >>>>>> Hi. >>>>>> >>>>>> I can find nothing really wrong in your configuration below. >>>>>> But, what happens if in this section : >>>>>> >>>>>> > <Connector port="8080" protocol="HTTP/1.1" >>>>>> > maxThreads="300" connectionTimeout="20000" >>>>>> > redirectPort="8443" /> >>>>>> >>>>>> you change the connectionTimeout to 3000 (= 3 seconds, instead of >>>>>> the above 20 seconds) ? >>>>>> >>>>>> Do you still see the number of threads remaining at the maximum ? >>>>>> >>>>>> See : >>>>>> http://tomcat.apache.org/tomcat-8.0-doc/config/http.html#Standard_ >>>>>> Im >>>>>> plementation >>>>>> --> connectionTimeout >>>>>> and the fact that it is also the default for keepAliveTimeout >>>>>> >>>>>> >>>>>> On 14.01.2017 07:30, smith wrote: >>>>>>> The server.xml: >>>>>>> >>>>>>> <?xml version='1.0' encoding='utf-8'?> >>>>>>> <!-- >>>>>>> Licensed to the Apache Software Foundation (ASF) under one >>>>>>> or more >>>>>>> contributor license agreements. See the NOTICE file >>>>>>> distributed with >>>>>>> this work for additional information regarding copyright >>>>>>> ownership. >>>>>>> The ASF licenses this file to You under the Apache License, >>>>>>> Version 2.0 >>>>>>> (the "License"); you may not use this file except in >>>>>>> compliance with >>>>>>> the License. You may obtain a copy of the License at >>>>>>> >>>>>>> http://www.apache.org/licenses/LICENSE-2.0 >>>>>>> >>>>>>> Unless required by applicable law or agreed to in writing, >>>>>>> software >>>>>>> distributed under the License is distributed on an "AS IS" >>>>>>> BASIS, >>>>>>> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either >>>>>>> express or implied. >>>>>>> See the License for the specific language governing >>>>>>> permissions and >>>>>>> limitations under the License. >>>>>>> --> >>>>>>> <!-- Note: A "Server" is not itself a "Container", so you may not >>>>>>> define subcomponents such as "Valves" at this level. >>>>>>> Documentation at /docs/config/server.html >>>>>>> --> >>>>>>> <Server port="8005" shutdown="SHUTDOWN"> >>>>>>> <Listener >>>>>>> className="org.apache.catalina.startup.VersionLoggerListener" /> >>>>>>> <!-- Security listener. Documentation at >>>>>>> /docs/config/listeners.html >>>>>>> <Listener >>>>>>> className="org.apache.catalina.security.SecurityListener" /> >>>>>>> --> >>>>>>> <!--APR library loader. Documentation at /docs/apr.html --> >>>>>>> <Listener >>>>>>> className="org.apache.catalina.core.AprLifecycleListener" >>>>>>> SSLEngine="on" /> >>>>>>> <!-- Prevent memory leaks due to use of particular >>>>>>> java/javax APIs--> >>>>>>> <Listener >>>>>>> className="org.apache.catalina.core.JreMemoryLeakPreventionListener" >>>>>>> /> >>>>>>> <Listener >>>>>>> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" >>>>>>> /> >>>>>>> <Listener >>>>>>> className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" >>>>>>> /> >>>>>>> >>>>>>> <!-- Global JNDI resources >>>>>>> Documentation at /docs/jndi-resources-howto.html >>>>>>> --> >>>>>>> <GlobalNamingResources> >>>>>>> <!-- Editable user database that can also be used by >>>>>>> UserDatabaseRealm to authenticate users >>>>>>> --> >>>>>>> <Resource name="UserDatabase" auth="Container" >>>>>>> type="org.apache.catalina.UserDatabase" >>>>>>> description="User database that can be updated >>>>>>> and saved" >>>>>>> factory="org.apache.catalina.users.MemoryUserDatabaseFactory" >>>>>>> pathname="conf/tomcat-users.xml" /> >>>>>>> </GlobalNamingResources> >>>>>>> >>>>>>> <!-- A "Service" is a collection of one or more >>>>>>> "Connectors" that share >>>>>>> a single "Container" Note: A "Service" is not itself >>>>>>> a "Container", >>>>>>> so you may not define subcomponents such as "Valves" >>>>>>> at this level. >>>>>>> Documentation at /docs/config/service.html >>>>>>> --> >>>>>>> <Service name="Catalina"> >>>>>>> >>>>>>> <!--The connectors can use a shared executor, you can >>>>>>> define one or more named thread pools--> >>>>>>> <!-- >>>>>>> <Executor name="tomcatThreadPool" >>>>>>> namePrefix="catalina-exec-" >>>>>>> maxThreads="150" minSpareThreads="4"/> >>>>>>> --> >>>>>>> >>>>>>> >>>>>>> <!-- A "Connector" represents an endpoint by which >>>>>>> requests are received >>>>>>> and responses are returned. Documentation at : >>>>>>> Java HTTP Connector: /docs/config/http.html >>>>>>> (blocking & non-blocking) >>>>>>> Java AJP Connector: /docs/config/ajp.html >>>>>>> APR (HTTP/AJP) Connector: /docs/apr.html >>>>>>> Define a non-SSL HTTP/1.1 Connector on port 8080 >>>>>>> --> >>>>>>> <Connector port="8080" protocol="HTTP/1.1" >>>>>>> maxThreads="300" connectionTimeout="20000" >>>>>>> redirectPort="8443" /> >>>>>>> <!-- A "Connector" using the shared thread pool--> >>>>>>> <!-- >>>>>>> <Connector executor="tomcatThreadPool" >>>>>>> port="8080" protocol="HTTP/1.1" >>>>>>> connectionTimeout="20000" >>>>>>> redirectPort="8443" /> >>>>>>> --> >>>>>>> <!-- Define a SSL HTTP/1.1 Connector on port 8443 >>>>>>> This connector uses the NIO implementation that >>>>>>> requires the JSSE >>>>>>> style configuration. When using the APR/native >>>>>>> implementation, the >>>>>>> OpenSSL style configuration is required as described >>>>>>> in the APR/native >>>>>>> documentation --> >>>>>>> <!-- >>>>>>> <Connector port="8443" >>>>>>> protocol="org.apache.coyote.http11.Http11NioProtocol" >>>>>>> maxThreads="150" SSLEnabled="true" >>>>>>> scheme="https" secure="true" >>>>>>> clientAuth="false" sslProtocol="TLS" /> >>>>>>> --> >>>>>>> >>>>>>> <!-- Define an AJP 1.3 Connector on port 8009 --> >>>>>>> <Connector port="8009" protocol="AJP/1.3" >>>>>>> redirectPort="8443" /> >>>>>>> >>>>>>> >>>>>>> <!-- An Engine represents the entry point (within >>>>>>> Catalina) that processes >>>>>>> every request. The Engine implementation for Tomcat >>>>>>> stand alone >>>>>>> analyzes the HTTP headers included with the request, >>>>>>> and passes them >>>>>>> on to the appropriate Host (virtual host). >>>>>>> Documentation at /docs/config/engine.html --> >>>>>>> >>>>>>> <!-- You should set jvmRoute to support load-balancing >>>>>>> via AJP ie : >>>>>>> <Engine name="Catalina" defaultHost="localhost" >>>>>>> jvmRoute="jvm1"> >>>>>>> --> >>>>>>> <Engine name="Catalina" defaultHost="localhost"> >>>>>>> >>>>>>> <!--For clustering, please take a look at documentation >>>>>>> at: >>>>>>> /docs/cluster-howto.html (simple how to) >>>>>>> /docs/config/cluster.html (reference documentation) >>>>>>> --> >>>>>>> <!-- >>>>>>> <Cluster >>>>>>> className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/> >>>>>>> --> >>>>>>> >>>>>>> <!-- Use the LockOutRealm to prevent attempts to guess >>>>>>> user passwords >>>>>>> via a brute-force attack --> >>>>>>> <Realm className="org.apache.catalina.realm.LockOutRealm"> >>>>>>> <!-- This Realm uses the UserDatabase configured in >>>>>>> the global JNDI >>>>>>> resources under the key "UserDatabase". Any edits >>>>>>> that are performed against this UserDatabase are >>>>>>> immediately >>>>>>> available for use by the Realm. --> >>>>>>> <Realm >>>>>>> className="org.apache.catalina.realm.UserDatabaseRealm" >>>>>>> resourceName="UserDatabase"/> >>>>>>> </Realm> >>>>>>> >>>>>>> <Host name="localhost" appBase="webapps" >>>>>>> unpackWARs="true" autoDeploy="true"> >>>>>>> >>>>>>> <!-- SingleSignOn valve, share authentication between >>>>>>> web applications >>>>>>> Documentation at: /docs/config/valve.html --> >>>>>>> <!-- >>>>>>> <Valve >>>>>>> className="org.apache.catalina.authenticator.SingleSignOn" /> >>>>>>> --> >>>>>>> >>>>>>> <!-- Access log processes all example. >>>>>>> Documentation at: /docs/config/valve.html >>>>>>> Note: The pattern used is equivalent to using >>>>>>> pattern="common" --> >>>>>>> >>>>>>> <Valve className="org.apache.catalina.valves.AccessLogValve" >>>>>>> directory="logs" >>>>>>> prefix="localhost_access_log" suffix=".txt" >>>>>>> pattern="%h,%t,%m,%U,%H,%s,%B,%D,%{User-Agent}i" /> >>>>>>> >>>>>>> <Context path="" allowLinking="true" >>>>>>> crossContext="true" docBase="/****/t" sessionCookieName="****" /> >>>>>>> </Host> >>>>>>> </Engine> >>>>>>> </Service> >>>>>>> </Server> >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: André Warnier (tomcat) [mailto:a...@ice-sa.com] >>>>>>> Sent: Friday, January 13, 2017 10:42 AM >>>>>>> To: users@tomcat.apache.org >>>>>>> Subject: Re: FW: tomcat 8080 thread not reduced >>>>>>> >>>>>>> On 13.01.2017 09:38, smith wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> From: smith [mailto:smith....@zoom.us] >>>>>>>> Sent: Tuesday, January 10, 2017 9:57 AM >>>>>>>> To: 'users' >>>>>>>> Subject: tomcat 8080 thread not reduced >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> We have installed Apache Tomcat/8.0.14, and found that after one >>>>>>>> period of time, the thread count for 8080(our port published) >>>>>>>> goes to 120 and never reduced even the busy count is only 3-4. >>>>>>>> >>>>>>>> Why? Tomcat8 not reduced the thread pool even the thread is >>>>>>>> idle, and the minSpareThreads for tomcat8 default is only 10. >>>>>>>> >>>>>>>> When will the thread reduce? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Best regards >>>>>>>> >>>>>>>> Smith >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> Hi. >>>>>>> >>>>>>> Please copy/paste your complete server.xml configuration file >>>>>>> (confidential things removed), so that we could have a useful >>>>>>> look at it. >>>>>>> Please edit *only* the confidential things, not entire sections. >>>>>>> Often, the issue is in the details. >>>>>>> >>>>>>> >>>>>>> >>>>>>> ----------------------------------------------------------------- >>>>>>> ---- >>>>>>> >>>>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>>>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>>>>> >>>>>>> >>>>>>> >>>>>>> ----------------------------------------------------------------- >>>>>>> ---- >>>>>>> >>>>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>>>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>>>>> >>>>>> >>>>>> >>>>>> ------------------------------------------------------------------ >>>>>> --- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>>>> >>>> >>>> >>>> -------------------------------------------------------------------- >>>> - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>> >>>> >>>> >>>> -------------------------------------------------------------------- >>>> - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>> >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>> For additional commands, e-mail: users-h...@tomcat.apache.org >>> >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>> For additional commands, e-mail: users-h...@tomcat.apache.org >>> >> >> > > Ce message et les fichiers d’accompagnement transmis avec celui-ci > s’adressent expressément au(x) destinataire(s) et peuvent contenir des > renseignements confidentiels et privilégiés. Si vous recevez ce message par > erreur, veuillez en aviser immédiatement l’expéditeur par courrier > électronique. Veuillez également ne pas en prendre connaissance et en > supprimer toutes les copies immédiatement. Technologies Interactives > Mediagrif Inc. et ses filiales n’acceptent aucune responsabilité à l’égard > des opinions exprimées dans le message ou des conséquences de tout virus > informatique qui pourrait être transmis avec ce message. Ce message fait > également l’objet d’un copyright. Il est interdit d’en reproduire, adapter ou > transmettre quelque partie que ce soit sans le consentement écrit du > détenteur du copyright. > > This email and any files transmitted with it are solely intended for the use > of the addressee(s) and may contain information that is confidential and > privileged. If you receive this email in error, please advise us by return > email immediately. Please also disregard the contents of the email, delete it > and destroy any copies immediately. Mediagrif Interactive Technologies Inc. > and its subsidiaries do not accept liability for the views expressed in the > email or for the consequences of any computer viruses that may be transmitted > with this email. This email is also subject to copyright. No part of it > should be reproduced, adapted or transmitted without the written consent of > the copyright owner. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
