Re: Websocket semaphore lock on close() blocks all tomcat threads
On 14/07/2020 21:08, Mark Thomas wrote: > On 14/07/2020 20:57, Sridhar Rao wrote: >> >> We notice a behavior with tomcat where it becomes unresponsive and all >> http threads go into a timed wait state and the node becomes unresponsive. >> >> Tomcat Version: 8.5.47 > > > >> Could this be a tomcat defect? > > Possibly. > > Let me take a look. I don't recall anything along these lines being > fixed since 8.5.47 but there might be. Yep. Looks like an issue in all current versions. Fixed for 10.0.x with back-ports to follow shortly. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Websocket semaphore lock on close() blocks all tomcat threads
On 14/07/2020 20:57, Sridhar Rao wrote: > > We notice a behavior with tomcat where it becomes unresponsive and all > http threads go into a timed wait state and the node becomes unresponsive. > > Tomcat Version: 8.5.47 > Could this be a tomcat defect? Possibly. Let me take a look. I don't recall anything along these lines being fixed since 8.5.47 but there might be. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: TCP connection vs Tomcat threads vs File Descriptors - please help
Thank you so much Rainer, for sparing time & answering my query. Vicky On Saturday, 17 October 2015 5:17 PM, Rainer Jung wrote: Am 17.10.2015 um 08:27 schrieb vicky: > Hi All, > can someone please help in understand that how TCP connections are > interlinked with the no. ofFile Descriptors & no of threads configured over a > machine . > Setup details :OD - Centos 6Tomcat7Java 7 > Recently i have faced an problem in while my application was having +20k TCP > connections in TIME-WAIT state resulting in choking my application though no > consumed threads & Files descriptors are pretty less than threshold. > 1) I would like to understand that is there any limit that how many TCP > connections a machine can open , IF YES then how to tune it . > 2) Second query , my understanding was every TCP connection will open one > file descriptor but in my currentsituation only 900 FD were used whereas the > TCP_WAIT connections where +20K .Kindly suggest how do i comprehend this . > How these are interlinked > 3) If i configure 600 threads in server.xml for my HTTP connector & if i'm > running that machine over 8 core cpu does that mean simultaneously 600 X 8 > (cpu core) =4800 threads will be served by my tomcat ? Let me give you an incomplete answer: A TCP connection in state TIME_WAIT does no longer exist from the point of view of the application/Tomcat/Java etc. So it does not need any application resources like threads. To understand TIME_WAIT, you should look for "TCP state diagram" in your favorite search engine or grab a copy of Steven's TCP/IP illustrated. You will find a picture like this: http://www.cs.northwestern.edu/~agupta/cs340/project2/TCPIP_State_Transition_Diagram.pdf (page 2) There you will see, that an ESTABLISHED connection can only enter TIME_WAIT state on the side of the connection, that first started the connection shut down by sending a FIN packet. And on that side it will always go through TIME_WAIT state. The default time during which a connection sits in TIME_WAIT on Linux seems to be 60, sometimes 120 seconds. So the total number of connections in that state is proportional to the number of connections per second that the local node starts closing. Example: Assume you run 100 new connections per second and all of the are closed by the local node first. That means in 60 seconds 6000 connections will pile up in state TIME_WAIT. In addition, removing TIME_WAIT connections from the OS list is not done continuously but in regular intervals, like e.g. every 5 seconds. So the real numbers can be slightly higher. Why are TIME_WAIT states bad? They don't need app resources, so why care? Because the increase the list of TCP connection states the OS has to manage and a huge number of such TIME_WAIT connections - a few 10.000s - can make the IP stack slower. The TIME_WAIT duration is not configurable for Linux only on some other Unixes. See the discussion at: http://comments.gmane.org/gmane.linux.network/244411 For some time you had to live with it and the only things you could do was - checking whether you could force more connections being closed by the remote side first - reducing the number of connections per second by increasing connection reuse, so keeping connections around for a longer time instead of constantly creating new ones. Both options would increase the need for app resources though, because the longer lifetime of established connections would often increase the number of threads needed. Now some people recommend using net.ipv4.tcp_tw_reuse, but that tunable seems to only apply to outgoing connections. Other suggest using net.ipv4.tcp_tw_recycle, but that one seems to make problems if clients sit behind a NAT device. See: http://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux.html Other people suggest tuning net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait: http://www.lognormal.com/blog/2012/09/27/linux-tcpip-tuning/ It could be, that this tunable will be replaced by nf_conntrack_tcp_timeout_time_wait in new kernels. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: TCP connection vs Tomcat threads vs File Descriptors - please help
Am 17.10.2015 um 08:27 schrieb vicky: Hi All, can someone please help in understand that how TCP connections are interlinked with the no. ofFile Descriptors & no of threads configured over a machine . Setup details :OD - Centos 6Tomcat7Java 7 Recently i have faced an problem in while my application was having +20k TCP connections in TIME-WAIT state resulting in choking my application though no consumed threads & Files descriptors are pretty less than threshold. 1) I would like to understand that is there any limit that how many TCP connections a machine can open , IF YES then how to tune it . 2) Second query , my understanding was every TCP connection will open one file descriptor but in my currentsituation only 900 FD were used whereas the TCP_WAIT connections where +20K .Kindly suggest how do i comprehend this . How these are interlinked 3) If i configure 600 threads in server.xml for my HTTP connector & if i'm running that machine over 8 core cpu does that mean simultaneously 600 X 8 (cpu core) =4800 threads will be served by my tomcat ? Let me give you an incomplete answer: A TCP connection in state TIME_WAIT does no longer exist from the point of view of the application/Tomcat/Java etc. So it does not need any application resources like threads. To understand TIME_WAIT, you should look for "TCP state diagram" in your favorite search engine or grab a copy of Steven's TCP/IP illustrated. You will find a picture like this: http://www.cs.northwestern.edu/~agupta/cs340/project2/TCPIP_State_Transition_Diagram.pdf (page 2) There you will see, that an ESTABLISHED connection can only enter TIME_WAIT state on the side of the connection, that first started the connection shut down by sending a FIN packet. And on that side it will always go through TIME_WAIT state. The default time during which a connection sits in TIME_WAIT on Linux seems to be 60, sometimes 120 seconds. So the total number of connections in that state is proportional to the number of connections per second that the local node starts closing. Example: Assume you run 100 new connections per second and all of the are closed by the local node first. That means in 60 seconds 6000 connections will pile up in state TIME_WAIT. In addition, removing TIME_WAIT connections from the OS list is not done continuously but in regular intervals, like e.g. every 5 seconds. So the real numbers can be slightly higher. Why are TIME_WAIT states bad? They don't need app resources, so why care? Because the increase the list of TCP connection states the OS has to manage and a huge number of such TIME_WAIT connections - a few 10.000s - can make the IP stack slower. The TIME_WAIT duration is not configurable for Linux only on some other Unixes. See the discussion at: http://comments.gmane.org/gmane.linux.network/244411 For some time you had to live with it and the only things you could do was - checking whether you could force more connections being closed by the remote side first - reducing the number of connections per second by increasing connection reuse, so keeping connections around for a longer time instead of constantly creating new ones. Both options would increase the need for app resources though, because the longer lifetime of established connections would often increase the number of threads needed. Now some people recommend using net.ipv4.tcp_tw_reuse, but that tunable seems to only apply to outgoing connections. Other suggest using net.ipv4.tcp_tw_recycle, but that one seems to make problems if clients sit behind a NAT device. See: http://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux.html Other people suggest tuning net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait: http://www.lognormal.com/blog/2012/09/27/linux-tcpip-tuning/ It could be, that this tunable will be replaced by nf_conntrack_tcp_timeout_time_wait in new kernels. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
TCP connection vs Tomcat threads vs File Descriptors - please help
Hi All, can someone please help in understand that how TCP connections are interlinked with the no. ofFile Descriptors & no of threads configured over a machine . Setup details :OD - Centos 6Tomcat7Java 7 Recently i have faced an problem in while my application was having +20k TCP connections in TIME-WAIT state resulting in choking my application though no consumed threads & Files descriptors are pretty less than threshold. 1) I would like to understand that is there any limit that how many TCP connections a machine can open , IF YES then how to tune it . 2) Second query , my understanding was every TCP connection will open one file descriptor but in my currentsituation only 900 FD were used whereas the TCP_WAIT connections where +20K .Kindly suggest how do i comprehend this . How these are interlinked 3) If i configure 600 threads in server.xml for my HTTP connector & if i'm running that machine over 8 core cpu does that mean simultaneously 600 X 8 (cpu core) =4800 threads will be served by my tomcat ?
Re: Tomcat threads dependency on net.core.somaxconn value on Linux
Thanks Christopher, it makes more sense now. On Fri, Apr 17, 2015 at 8:51 PM, Christopher Schultz < ch...@christopherschultz.net> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Satish, > > On 4/17/15 7:20 AM, satish jupalli wrote: > > I would like to support a bust in traffic we are planning to > > increase the umber of threads by adding executor pool with > > misSpareThread to 100 and accept count to 5. However, I'm > > trying to understand the correlation between the tomcat connector > > acceptCount and ThreadCount to the Linux config > > net.core.somaxconn(number of incoming connections) which defaults > > to 128. > > The thread count isn't really a part of this discussion: only the > accept count and /proc/sys/net/core/somaxx > > > Does it makes sense to increase the thread count alone with out > > increasing net.core.somaxconn value? > > Probably not; Linux limits the accept queue using that value and will > not allow client code to exceed that limit. > > https://computing.llnl.gov/linux/slurm/high_throughput.html > > > Below is the config that we are planning to use on Tomcat 7.0.42 on > > Linux 5.x. > > (You should upgrade to 7.0.62 as soon as it feasible for you.) > > > > minSpareThreads="100" maxThreads="300"/> > > > > > connectionTimeout="2" redirectPort="8443" > > enableLookups="false" keepAliveTimeout="30" > > maxKeepAliveRequests="1" acceptCount="5" socketBuffer="10240" > > executor="tomcatThreadPool" compression="on" > > compressableMimeType="text/html, text/xml" xpoweredBy="false" > > server="false" /> > > > > Any inputs regarding the Tomcat threads and Linux > > net.core.somaxconn will be greatly appreciated to support many > > concurrent connections. > > If you use acceptCount="5", Tomcat will ask for that socket queue > size and simply won't get it. You should definitely raise the value of > somaxx on Linux. > > - -chris > -BEGIN PGP SIGNATURE- > Version: GnuPG v2 > Comment: GPGTools - http://gpgtools.org > > iQIcBAEBCAAGBQJVMQG4AAoJEBzwKT+lPKRYFuMQAMMbBv7v5jOTHz302TMYNbMB > sc1qH9IuV7Z1DrqAOE+yN19xdzt+7cakNl5YWomHr/U3ePNianLeqY+27WYYUm9V > gTw/kHqBOUYfWttnlFzDCBidUDPw6m3TnhFq8Oia5UBdfPh4IQDR+zxF9FCkncUi > 51evAzsp4gzAOMjDUmxMQMCfYQML1l+VX56Za4RT4S6z2L6HCxyjeVYQSc2nsLnU > Y6G3X0ccomxszHhb5GuU82lldgKjw5BDubkp57/fxBajZ6QcWSU7sr+HlaoAUfLz > +p/PSJhyXDMxlKqGmGsBK8BMMsK5H26yup5LtpKkLDEJuUYSv5N1rsklmQCbUuXk > 0pBzFmVKQG2CRh2miutBC7Vr5l3AOP5ghV2uOOvuCBg+Sg8pzImase2m9Nw9Z1sh > XnLrrJq7a/nvd4C2jRT/y01AJNfv0qJG95RjBxoVdoQ629fjK05Z2MB7avL2Qzwq > 8JpcwhkfhVAWW87JVAIMPluILNuhCCg/LDQSVj4twdtGBy181E29yNSfmIRkq33j > 2Jc83/gLgNU+maM4RzXKWzFYpk8ccLMhHsbh8txqswK+rV8XDfQTLZzxy43IawCK > Nc1rBnWCVQOPMxjumfZWuZtdAy/HlSpw1otRVkXNVONOCLwMcAggLbjeO1TFJiJj > MArVB8ONarkbkDts2quG > =vSf+ > -END PGP SIGNATURE- > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: Tomcat threads dependency on net.core.somaxconn value on Linux
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Satish, On 4/17/15 7:20 AM, satish jupalli wrote: > I would like to support a bust in traffic we are planning to > increase the umber of threads by adding executor pool with > misSpareThread to 100 and accept count to 5. However, I'm > trying to understand the correlation between the tomcat connector > acceptCount and ThreadCount to the Linux config > net.core.somaxconn(number of incoming connections) which defaults > to 128. The thread count isn't really a part of this discussion: only the accept count and /proc/sys/net/core/somaxx > Does it makes sense to increase the thread count alone with out > increasing net.core.somaxconn value? Probably not; Linux limits the accept queue using that value and will not allow client code to exceed that limit. https://computing.llnl.gov/linux/slurm/high_throughput.html > Below is the config that we are planning to use on Tomcat 7.0.42 on > Linux 5.x. (You should upgrade to 7.0.62 as soon as it feasible for you.) > minSpareThreads="100" maxThreads="300"/> > > connectionTimeout="2" redirectPort="8443" > enableLookups="false" keepAliveTimeout="30" > maxKeepAliveRequests="1" acceptCount="5" socketBuffer="10240" > executor="tomcatThreadPool" compression="on" > compressableMimeType="text/html, text/xml" xpoweredBy="false" > server="false" /> > > Any inputs regarding the Tomcat threads and Linux > net.core.somaxconn will be greatly appreciated to support many > concurrent connections. If you use acceptCount="5", Tomcat will ask for that socket queue size and simply won't get it. You should definitely raise the value of somaxx on Linux. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v2 Comment: GPGTools - http://gpgtools.org iQIcBAEBCAAGBQJVMQG4AAoJEBzwKT+lPKRYFuMQAMMbBv7v5jOTHz302TMYNbMB sc1qH9IuV7Z1DrqAOE+yN19xdzt+7cakNl5YWomHr/U3ePNianLeqY+27WYYUm9V gTw/kHqBOUYfWttnlFzDCBidUDPw6m3TnhFq8Oia5UBdfPh4IQDR+zxF9FCkncUi 51evAzsp4gzAOMjDUmxMQMCfYQML1l+VX56Za4RT4S6z2L6HCxyjeVYQSc2nsLnU Y6G3X0ccomxszHhb5GuU82lldgKjw5BDubkp57/fxBajZ6QcWSU7sr+HlaoAUfLz +p/PSJhyXDMxlKqGmGsBK8BMMsK5H26yup5LtpKkLDEJuUYSv5N1rsklmQCbUuXk 0pBzFmVKQG2CRh2miutBC7Vr5l3AOP5ghV2uOOvuCBg+Sg8pzImase2m9Nw9Z1sh XnLrrJq7a/nvd4C2jRT/y01AJNfv0qJG95RjBxoVdoQ629fjK05Z2MB7avL2Qzwq 8JpcwhkfhVAWW87JVAIMPluILNuhCCg/LDQSVj4twdtGBy181E29yNSfmIRkq33j 2Jc83/gLgNU+maM4RzXKWzFYpk8ccLMhHsbh8txqswK+rV8XDfQTLZzxy43IawCK Nc1rBnWCVQOPMxjumfZWuZtdAy/HlSpw1otRVkXNVONOCLwMcAggLbjeO1TFJiJj MArVB8ONarkbkDts2quG =vSf+ -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Tomcat threads dependency on net.core.somaxconn value on Linux
Hi, I would like to support a bust in traffic we are planning to increase the umber of threads by adding executor pool with misSpareThread to 100 and accept count to 5. However, I'm trying to understand the correlation between the tomcat connector acceptCount and ThreadCount to the Linux config net.core.somaxconn(number of incoming connections) which defaults to 128. Does it makes sense to increase the thread count alone with out increasing net.core.somaxconn value? Below is the config that we are planning to use on Tomcat 7.0.42 on Linux 5.x. Any inputs regarding the Tomcat threads and Linux net.core.somaxconn will be greatly appreciated to support many concurrent connections. Regards Satish Jupalli
Re: Running native JNI calls in parallel from different Tomcat threads
On Jan 5, 2014, at 8:23 PM, Jasmeet Chhabra wrote: > Hi, > I have a Tomcat server that receives many web requests in parallel. The > web requests make a native call using JNI to my library. These calls are > long running (400 ms each call). What I have noticed is that even though I > am getting web requests in parallel, the native calls seem to be > serialized. I don't take any locks in the native code that would force > serialization. Is there anything in Tomcat that prevents same native calls > being called in parallel from separate Tomcat threads? Seems unlikely to me. Perhaps you can give us some basic info about you setup. Start with your JVM, Tomcat version and configuration (minus comments and sensitive info). Also, try taking some thread dumps while the server is busy. Not sure it'll show everything, but it could give you some clues. Dan > > Thanks, > Jas - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Running native JNI calls in parallel from different Tomcat threads
Hi, I have a Tomcat server that receives many web requests in parallel. The web requests make a native call using JNI to my library. These calls are long running (400 ms each call). What I have noticed is that even though I am getting web requests in parallel, the native calls seem to be serialized. I don't take any locks in the native code that would force serialization. Is there anything in Tomcat that prevents same native calls being called in parallel from separate Tomcat threads? Thanks, Jas
Re: Tomcat Threads busy
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Anil, On 7/23/13 4:35 AM, Anil Goyal -X (anigoyal - ARICENT TECHNOLOGIES MAURIITIUS LIMITED at Cisco) wrote: That logic needs to happen in another layer, in a load-balancer. Tomcat cannot do this on its own. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJR7q9aAAoJEBzwKT+lPKRYyWsQAMCXavCXXodFgfPMaWg000eQ 0s4i4zVe3fqcUjfgeMoVua1xmYvV8cfe1t+4GDdZWDqBIpc+fBCfi0pkQKa0itUL vmQ+vKQ/QJ9U+v0IJ272Hj/C7ar0F5/7m9A3SDF+dMxHPHwa6skUOG2X4RQXLzah 8KZ9aGQUigsMkwQLtV6Oa3eAkqxknyLPyyuMbrEWNLj7rcQz/6lghT2nWI/SdKz0 OTOGw+t4U01JM9cWldxqt2oJ7Q3/r8sVDhBcEURJPGXXs3g6SoOOJl0kZb8a5QQj 5tNSlb/jKx0ACT4jyUQqQOUx/flksa8tIjio4AWwgWRklqgF6xW3S8tGfxq1inRm PL33KkkBeV7VyoTQiw9QRash37wx2uuGTP+BC8ARO8zu8ugF66YxeNBmBcQEXipX vSthyIxvTMqf0JBGAPardKn2eYlCeV2lkvGjiVwdq1J6DlB4eaBaXj5JiOh+sy1O DMRCAjHj4G0QNOBCFdw+jcXRT1LIidRxjn8yRvc5jql13jBLf80A/mIHWJudOThA ox7lxyzu3V+G1iCfKdgIt7tO+QtjJiYpSSTDmRF253GvYGF4W4KxIOpYPrQy/ZVH vG4P9BY4JEkXNPQKpCI0zzc31GyhP+D2yHSPLmJCTTzsX9bhRuYfE4ROzui0fPxh LRb9RuK/f/hfaqJCixPt =VwAB -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Tomcat Threads busy
Hi when a request comes to tomcat and tomcat does not have any free thread available or all the threads are busy and request queue is also full then where I need to write the logic for redirect the request to some other server. I have ConnectionRedirectionValve.java already written but in that file the request redirection logic is handled when threads are not sufficient at application level means application inside tomcat do not have any spare semaphore that is handled in this file. Even though when tomcat does not have sufficient threads then I do not think the flow will move to ConnectionRedirectionValve.java So please suggest how to handle this Thanks Anil
Re: Tomcat threads management
2011/4/29 André Warnier : > Server version: Apache Tomcat/5.5 > Server built: Oct 15 2008 12:57:44 > Server number: 5.5.26.0 > > Intuitively, I would expect that after some time, the number of live threads > would be decreasing back to the minSpareThreads value, but that does not > seem to be happening. > The number of threads stays at the same level of about 100, apparently > forever (at least several hours by now). > Is this normal ? In short: you have to define an Executor -- only it has the ability to decrease the number of working threads over time. - That is true for Tomcat 6 and later. (You are using Tomcat 5.5 and I just do not remember how it goes there. IIRC, thread pool in TC 6 was simplified wrt. to 5.5 one). Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Tomcat threads management
Hi. versions : Front-end httpd : [Mon Apr 25 19:13:18 2011] [notice] Apache/2.2.9 (Debian) DAV/2 mod_jk/1.2.26 mod_apreq2-20051231/2.6.0 mod_perl/2.0.4 Perl/v5.10.0 configured -- resuming normal operations Tomcat: Using CATALINA_BASE: /usr/share/tomcat5.5 Using CATALINA_HOME: /usr/share/tomcat5.5 Using CATALINA_TMPDIR: /usr/share/tomcat5.5/temp Using JRE_HOME: /usr/lib/jvm/java-6-sun/jre Server version: Apache Tomcat/5.5 Server built: Oct 15 2008 12:57:44 Server number: 5.5.26.0 OS Name:Linux OS Version: 2.6.26-2-amd64 Architecture: amd64 JVM Version:1.6.0_22-b04 JVM Vendor: Sun Microsystems Inc. vogon2:/usr/share/tomcat5.5/bin# (both installed from standard Debian package) in server.xml : Note that there is no "maxSpareThreads" attribute, which according to the documentation would result in a default value of 50. Question : With the above settings for the AJP connector, I run some stress test using the Apache "ab" program to access a Tomcat application, with a concurrency of 100 simultaneous requests. After the test finishes, the Tomcat Manager's server status shows the situation below, with 98 threads alive. There is only this one test application installed, and after the test above there are no more accesses to Tomcat. Intuitively, I would expect that after some time, the number of live threads would be decreasing back to the minSpareThreads value, but that does not seem to be happening. The number of threads stays at the same level of about 100, apparently forever (at least several hours by now). Is this normal ? How does Tomcat (or the AJP Connector) manage these threads ? Should idle ones not disappear after a while ? (Accessorily, the display below shows "Min spare threads: 0" and "Max spare threads: 0", which does not match the Connector setting) Tomcat Manager output : ajp-8009 Max threads: 150 Min spare threads: 0 Max spare threads: 0 Current thread count: 98 Current thread busy: 1 Keeped alive sockets count: 0 Max processing time: 293263 ms Processing time: 21367.768 s Request count: 15433 Error count: 3 Bytes received: 7.47 MB Bytes sent: 371.22 MB Stage TimeB Sent B Recv Client VHost Request R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? S 1 ms0 KB0 KBx.x.x.x host.company.comGET /manager/status/all HTTP/1.1 R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ? ? ? ? ? R ? ?
AW: Restarting Tomcat & Threads
Mit freundlichen Grüßen Thomas Strauß Geschäftsführer Entwicklung SRS PaperDynamixR DIGITAL SCHON AUF DEM PAPIER SRS-Management GmbH Berliner Ring 93 64625 Bensheim T +49 6251 85 424 - 20 F +49 6251 85 424 - 14 M +49 174 2110912 www.srs-management.de www.srs-paperdynamix.de HRB 25262 AG Darmstadt Geschäftsführer: Detlev Homilius, Thomas Strauß > -Ursprüngliche Nachricht- > Von: David kerber [mailto:dcker...@verizon.net] > Gesendet: Dienstag, 19. April 2011 17:02 > An: Tomcat Users List > Betreff: Re: Restarting Tomcat & Threads > > On 4/19/2011 10:34 AM, Zbynek Vavros wrote: > > > > - How do you kill Tomcat ? > > > > I stop its service. > > > > - You say "after start" and "I didnt [sic] start anything" in the > same > > sentence; you're contradicting yourself. > > > > I start Tomcat which starts my webapp but connection to Lotus is > started > > separatedlly. > > = I start my webapp but do not start "Lotus connector". > > > > 1) My code(thread) could not be executed unless I start it via web > > interface.Which I dont. > > > > 2) Im talking about log from log4j so it is log of my application.No > one > > else's. > > So you have to go through that web interface to start the lotus thread, > but you're not doing that and the thread is still starting up? > > Sounds like the method that starts the thread is being called from > somewhere that you're not expecting. Any chance it's being fired off > from a timer or a listener? > Maybe you are running the Lotus connector through a servlet that defines "load-on-startup"? > > > > > > > > Zbynek > > > > Kind Regards / Mit > > freundlichen Grüßen / > > Üdvözlettel / S > > pozdravem: > > > > > > Zbynek VAVROS > (Embedded > > Development image > moved > > to > file: > > > pic01634.gif) > > > > Delivery Centre 616 00, Brno > >Central Europe Technicka 21 > > Brno SITE Czech Republic > > > > (Embedded image moved to > > file: pic31451.gif)Phone: 420-53341- x6283 > >Mobile: > >E-mail: zbynek_vav...@cz.ibm.com > > > > > > > > > > IBM Global Services > >Delivery Center Czech > > Republic, s.r.o. > > Registered address: > > Brno, Technicka 2995/21, > > Zip code: 61600, Company > > ID: 26244535 > >Entered in the > > Commercial Register > > maintained by the > >Regional Court in Brno > >(Part C, Entry 39922) > > > > IBM Global Services > >Delivery Center Czech > > Republic, s.r.o. > >Sídlo: Brno, Technická > >2995/21, PSČ 61600 IČ: > > 26244535 > > Zapsaná v obchodním > > rejstříku, vedeném > >Krajským soudem v Brně > >oddíl C, vlozka 39922 > > > > > > > > > > > > > > |> > > | From: | > > |> > >>- > --- > --| > >|"Caldarale, Charles R" > | > >>- > --- > --| > > |> > > | To:| > > |> > >>- > --- > --| > >|Tomcat Users List > | > >>- > ----------- > --| > > |> > > | Date: | > > |> > >>- > ------- > --| > >|04/19/2011 04:09 PM > | > >>- > --- > --| > > |> > > | Subject: | > > |&
Re: Restarting Tomcat & Threads
On 4/19/2011 10:34 AM, Zbynek Vavros wrote: - How do you kill Tomcat ? I stop its service. - You say "after start" and "I didnt [sic] start anything" in the same sentence; you're contradicting yourself. I start Tomcat which starts my webapp but connection to Lotus is started separatedlly. = I start my webapp but do not start "Lotus connector". 1) My code(thread) could not be executed unless I start it via web interface.Which I dont. 2) Im talking about log from log4j so it is log of my application.No one else's. So you have to go through that web interface to start the lotus thread, but you're not doing that and the thread is still starting up? Sounds like the method that starts the thread is being called from somewhere that you're not expecting. Any chance it's being fired off from a timer or a listener? Zbynek Kind Regards / Mit freundlichen Grüßen / Üdvözlettel / S pozdravem: Zbynek VAVROS (Embedded Development image moved to file: pic01634.gif) Delivery Centre 616 00, Brno Central Europe Technicka 21 Brno SITE Czech Republic (Embedded image moved to file: pic31451.gif)Phone: 420-53341- x6283 Mobile: E-mail: zbynek_vav...@cz.ibm.com IBM Global Services Delivery Center Czech Republic, s.r.o. Registered address: Brno, Technicka 2995/21, Zip code: 61600, Company ID: 26244535 Entered in the Commercial Register maintained by the Regional Court in Brno (Part C, Entry 39922) IBM Global Services Delivery Center Czech Republic, s.r.o. Sídlo: Brno, Technická 2995/21, PSČ 61600 IČ: 26244535 Zapsaná v obchodním rejstříku, vedeném Krajským soudem v Brně oddíl C, vlozka 39922 |> | From: | |> >--| |"Caldarale, Charles R" | >--| |> | To:| |> >--| |Tomcat Users List | >--| |> | Date: | |> >--| |04/19/2011 04:09 PM | >--| |> | Subject: | |> >----------| |RE: Restarting Tomcat& Threads | >--------------| From: Zbynek Vavros [mailto:zbynek_vav...@cz.ibm.com] Subject: Restarting Tomcat& Threads I have noticed that when I kill Tomcat (shut down its service) How do you kill Tomcat? (Be specific.) then after start this thread is started automatically (I can see messages about connecting to Lotus in log when I didnt start anything). You say "after start" and "I didnt [sic] start anything" in the same sentence; you're contradicting yourself. Tomcat knows nothing about your thread or the connection to Lotus, so one of two things is happening: 1) Your code is being executed, you just aren't aware of it. 2) Something else is connecting to Lotus and you're mistaking that for your code. - Chuck - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Restarting Tomcat & Threads
Zbynek Vavros wrote: - How do you kill Tomcat ? I stop its service. - You say "after start" and "I didnt [sic] start anything" in the same sentence; you're contradicting yourself. I start Tomcat which starts my webapp but connection to Lotus is started separatedlly. = I start my webapp but do not start "Lotus connector". 1) My code(thread) could not be executed unless I start it via web interface.Which I dont. 2) Im talking about log from log4j so it is log of my application.No one else's. The tomcat code, by itself, has no reason to connect to Lotus. So if something connects to Lotus, it must be in the code of your application. But you say above that you do not do that. So something above is wrong. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Restarting Tomcat & Threads
---| |> | Date: | |> >--| |04/19/2011 04:09 PM | >--| |> | Subject: | |> >--------------| |RE: Restarting Tomcat & Threads | >----------| > From: Zbynek Vavros [mailto:zbynek_vav...@cz.ibm.com] > Subject: Restarting Tomcat & Threads > I have noticed that when I kill Tomcat (shut down its service) How do you kill Tomcat? (Be specific.) > then after start this thread is started automatically (I can > see messages about connecting to Lotus in log when I didnt > start anything). You say "after start" and "I didnt [sic] start anything" in the same sentence; you're contradicting yourself. Tomcat knows nothing about your thread or the connection to Lotus, so one of two things is happening: 1) Your code is being executed, you just aren't aware of it. 2) Something else is connecting to Lotus and you're mistaking that for your code. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Restarting Tomcat & Threads
> From: Zbynek Vavros [mailto:zbynek_vav...@cz.ibm.com] > Subject: Restarting Tomcat & Threads > I have noticed that when I kill Tomcat (shut down its service) How do you kill Tomcat? (Be specific.) > then after start this thread is started automatically (I can > see messages about connecting to Lotus in log when I didnt > start anything). You say "after start" and "I didnt [sic] start anything" in the same sentence; you're contradicting yourself. Tomcat knows nothing about your thread or the connection to Lotus, so one of two things is happening: 1) Your code is being executed, you just aren't aware of it. 2) Something else is connecting to Lotus and you're mistaking that for your code. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Restarting Tomcat & Threads
Hi, I would like to ask on strange behavior Im experiencing. I have webapp that connects to Lotus Notes periodically(every 1 minute). For this I have class that implements Runnable that takes care about connection itself. Im starting this using basic : Thread t = new Thread(MyClassImplementingRunnable); t.start(); I have noticed that when I kill Tomcat (shut down its service) then after start this thread is started automatically (I can see messages about connecting to Lotus in log when I didnt start anything). Is this usual behavior? Is there any was for threads "not to start automatically" ? Im beginner with Tomcat so excuse my lack of knowledge. Thanks Zbynek Kind Regards / Mit freundlichen Grüßen / Üdvözlettel / S pozdravem: Zbynek VAVROS (Embedded Development image moved to file: pic31252.gif) Delivery Centre 616 00, Brno Central Europe Technicka 21 Brno SITE Czech Republic (Embedded image moved to file: pic21477.gif)Phone: 420-53341- x6283 Mobile: E-mail: zbynek_vav...@cz.ibm.com IBM Global Services Delivery Center Czech Republic, s.r.o. Registered address: Brno, Technicka 2995/21, Zip code: 61600, Company ID: 26244535 Entered in the Commercial Register maintained by the Regional Court in Brno (Part C, Entry 39922) IBM Global Services Delivery Center Czech Republic, s.r.o. Sídlo: Brno, Technická 2995/21, PSČ 61600 IČ: 26244535 Zapsaná v obchodním rejstříku, vedeném Krajským soudem v Brně oddíl C, vlozka 39922 - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat threads
Thanks Folks. On Wed, Mar 3, 2010 at 1:03 PM, Peter Crowther wrote: > On 3 March 2010 18:24, Bharath Vasudevan wrote: > > > Hmmm... > > > > "No, the server will allocate maxThreads request handlers; the other > > requests would sit in the TCP stack's queue (not in the JVM), up to the > > configured acceptCount value - which you can set as high as your OS > > allows." > > > > I was assuming that the tomcat main thread is going to pick it up from > the > > TCP stack and let the web server thread handle the request. Wont tomcat > > reject the request when max threads have been reached (since it would > have > > already picked up tcp layer)? > > > > No, because the container doesn't know when a thread might become > available. Serving content after a delay is considered better than not > serving content at all, so the requests are simply queued in the TCP stack. > > - Peter >
Re: Tomcat threads
On 3 March 2010 18:24, Bharath Vasudevan wrote: > Hmmm... > > "No, the server will allocate maxThreads request handlers; the other > requests would sit in the TCP stack's queue (not in the JVM), up to the > configured acceptCount value - which you can set as high as your OS > allows." > > I was assuming that the tomcat main thread is going to pick it up from the > TCP stack and let the web server thread handle the request. Wont tomcat > reject the request when max threads have been reached (since it would have > already picked up tcp layer)? > > No, because the container doesn't know when a thread might become available. Serving content after a delay is considered better than not serving content at all, so the requests are simply queued in the TCP stack. - Peter
Re: Tomcat threads
Hmmm... "No, the server will allocate maxThreads request handlers; the other requests would sit in the TCP stack's queue (not in the JVM), up to the configured acceptCount value - which you can set as high as your OS allows." I was assuming that the tomcat main thread is going to pick it up from the TCP stack and let the web server thread handle the request. Wont tomcat reject the request when max threads have been reached (since it would have already picked up tcp layer)? On Wed, Mar 3, 2010 at 9:45 AM, Caldarale, Charles R < chuck.caldar...@unisys.com> wrote: > > From: Bharath Vasudevan [mailto:bharath@gmail.com] > > Subject: Re: Tomcat threads > > > > When tomcat gets a request, it does a socket send to some > > other process to handle the request and then respond. > > Tomcat doesn't do that - your webapp does. > > You have now introduced a previously unmentioned dependency - that the > response depends on the behavior of an external resource, unrelated to > Tomcat. That changes the picture considerably. The neat thing the servlet > 3.0 AsyncContext does is handle this case by letting you decouple the > original Request and Response objects from the original processing thread. > > > But assuming 20k client requests come in at the same time, > > the server would try to allocate 20k threads and handle it. > > No, the server will allocate maxThreads request handlers; the other > requests would sit in the TCP stack's queue (not in the JVM), up to the > configured acceptCount value - which you can set as high as your OS allows. > > > Mostly the system would trash and go down. > > Which is my point about why you don't want to set maxThreads to arbitrarily > large (or small) values. > > > But if it was to be handled gracefully, we can have the > > threads pick up the request throw it in a queue and let > > the worker threads work and respond. > > Which you can do in your webapp by sending an interim response to the > client and having the client poll for the final one. You'd have to disable > keep-alives as well. The 3.0 spec allows the container to perform this > task, rather than requiring the webapp to do so. > > - Chuck > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is thus for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all computers. > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: Tomcat threads
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Bharath, On 3/3/2010 12:21 PM, Bharath Vasudevan wrote: > Let me explain the scenario. When tomcat gets a request, it does a socket > send to some other process to handle the request and then respond. This > would happen fast. But assuming 20k client requests come in at the same > time, the server would try to allocate 20k threads and handle it. Mostly the > system would trash and go down. > > But if it was to be handled gracefully, we can have the threads pick up the > request throw it in a queue and let the worker threads work and respond. Let > me know if I am missing something here. Wouldn't you have to limit that queue, otherwise you might never catch up with the requests and run out of memory? Limiting the work queue size versus the thread pool size are equivalent in terms of user experience: at some point, the remote requests are refused. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkuOpKMACgkQ9CaO5/Lv0PB/GQCgk+dKujEGRKF69ZVjmn7uWcVA rbAAniTg+2nvvOygxACU1tNCmRauitsS =wO/z -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat threads
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Bharath, On 3/2/2010 7:42 PM, Bharath Vasudevan wrote: > Why is it illlogical? Fast is a relative term. If the number of requests > increases, the number of threads that can be handled by the system goes down > . The context switches and the pain to handle the switches makes handling of > the requests in lesser threads which is scalable. If you can afford the memory and CPU to create a second thread pool that does the "real work" of your webapp, why not just allocate those same threads in the request processing thread pool? Seems like a wash in terms of thread availability with a significant reduction in overhead of managing multiple thread pools, passing data between them, etc. and trying to work around a problem that really doesn't exist. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkuOpC4ACgkQ9CaO5/Lv0PCGXACgkVGoBYzmZwMUPGT9ZE5G8vlD LqEAn1IX6EuxlfY4hPZGO5uJOzbb2gI8 =pvO1 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat threads
> From: Bharath Vasudevan [mailto:bharath@gmail.com] > Subject: Re: Tomcat threads > > Looks like comet can asynchronously push data to the user. You do understand that there really is no such thing as "push" (at least in HTTP)? The client has to have a receive up, so you can't use just a plain browser - the client has to be Comet-compatible. With Tomcat 7, you will be able to use the trick Bill B mentioned: send incomplete chunked responses to keep the client from timing out until the full response is ready, without dedicating a thread to the task. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat threads
> From: Bharath Vasudevan [mailto:bharath@gmail.com] > Subject: Re: Tomcat threads > > When tomcat gets a request, it does a socket send to some > other process to handle the request and then respond. Tomcat doesn't do that - your webapp does. You have now introduced a previously unmentioned dependency - that the response depends on the behavior of an external resource, unrelated to Tomcat. That changes the picture considerably. The neat thing the servlet 3.0 AsyncContext does is handle this case by letting you decouple the original Request and Response objects from the original processing thread. > But assuming 20k client requests come in at the same time, > the server would try to allocate 20k threads and handle it. No, the server will allocate maxThreads request handlers; the other requests would sit in the TCP stack's queue (not in the JVM), up to the configured acceptCount value - which you can set as high as your OS allows. > Mostly the system would trash and go down. Which is my point about why you don't want to set maxThreads to arbitrarily large (or small) values. > But if it was to be handled gracefully, we can have the > threads pick up the request throw it in a queue and let > the worker threads work and respond. Which you can do in your webapp by sending an interim response to the client and having the client poll for the final one. You'd have to disable keep-alives as well. The 3.0 spec allows the container to perform this task, rather than requiring the webapp to do so. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat threads
That would be the last go. Trying to figure out if there is techniques to handle such scenarios. Looks like comet can asynchronously push data to the user. If this is going to be seamless to the client, it might as well look like a response for their request. On Wed, Mar 3, 2010 at 9:25 AM, Mark Thomas wrote: > On 03/03/2010 17:21, Bharath Vasudevan wrote: > > Hi Charles, > > > > Let me explain the scenario. When tomcat gets a request, it does a socket > > send to some other process to handle the request and then respond. This > > would happen fast. But assuming 20k client requests come in at the same > > time, the server would try to allocate 20k threads and handle it. Mostly > the > > system would trash and go down. > > > > But if it was to be handled gracefully, we can have the threads pick up > the > > request throw it in a queue and let the worker threads work and respond. > Let > > me know if I am missing something here. > > Like the fact that Tomncat is open source and you could just look at the > source code for the various connectors and figure this all out for > yourself? You don't even have to read the source. The connectors > documentation covers this pretty well. > > Mark > > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: Tomcat threads
Thanks Bill. Comet is something that I can dig into :). On Tue, Mar 2, 2010 at 7:21 PM, Bill Barker wrote: > > > "Caldarale, Charles R" wrote in message > news:99c8b2929b39c24493377ac7a121e21f96cb817...@usea-exch8.na.uis.unisys.com. > .. > > From: Bharath Vasudevan [mailto:bharath....@gmail.com] >>> Subject: Re: Tomcat threads >>> >>> If we get a request on a thread, let some other thread do >>> the work for it and store the response info. The thread >>> which does the work writes the response on that request. >>> >> >> If the processing is fast, why would you go to the complexity and overhead >> of switching to another thread to process the request? >> >> You should also read the servlet spec, in particular SRV.2.3.3.3: >> >> "Implementations of the request and response objects are not guaranteed to >> be thread >> safe. This means that they should only be used within the scope of the >> request handling >> thread. >> >> "References to the request and response objects should not be given to >> objects >> executing in other threads as the resulting behavior may be >> nondeterministic. If >> the thread created by the application uses the container-managed objects, >> such as >> the request or response object, those objects must be accessed only within >> the >> servlet's service life cycle and such thread itself should have a life >> cycle within >> the life cycle of the servlet's service method because accessing those >> objects >> after the service method ends may cause undeterministic problems." >> >> The illogical behavior you're asking for simply isn't allowed. >> >> > At least not until Tomcat 7 ;). In there, you can do exactly what the user > is asking for using the AsyncContext. The details are in the Servlet 3.0 > spec, but basically the request is handed off to another thread and the > original one returns to the pool. This also allows for two-way > communication instead of relying on polling. > > Of course Comet offers similar functionality in Tomcat 6 if you don't mind > being bound to Tomcat. > > > - Chuck >> >> >> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY >> MATERIAL and is thus for use only by the intended recipient. If you received >> this in error, please contact the sender and delete the e-mail and its >> attachments from all computers. >> > > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: Tomcat threads
On 03/03/2010 17:21, Bharath Vasudevan wrote: > Hi Charles, > > Let me explain the scenario. When tomcat gets a request, it does a socket > send to some other process to handle the request and then respond. This > would happen fast. But assuming 20k client requests come in at the same > time, the server would try to allocate 20k threads and handle it. Mostly the > system would trash and go down. > > But if it was to be handled gracefully, we can have the threads pick up the > request throw it in a queue and let the worker threads work and respond. Let > me know if I am missing something here. Like the fact that Tomncat is open source and you could just look at the source code for the various connectors and figure this all out for yourself? You don't even have to read the source. The connectors documentation covers this pretty well. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat threads
Hi Charles, Let me explain the scenario. When tomcat gets a request, it does a socket send to some other process to handle the request and then respond. This would happen fast. But assuming 20k client requests come in at the same time, the server would try to allocate 20k threads and handle it. Mostly the system would trash and go down. But if it was to be handled gracefully, we can have the threads pick up the request throw it in a queue and let the worker threads work and respond. Let me know if I am missing something here. Bharath On Tue, Mar 2, 2010 at 7:22 PM, Caldarale, Charles R < chuck.caldar...@unisys.com> wrote: > > From: Bharath Vasudevan [mailto:bharath@gmail.com] > > Subject: Re: Tomcat threads > > > > Why is it illlogical? > > 40+ years of system architecture experience. > > > If the number of requests increases, the number of threads > > that can be handled by the system goes down. > > You'll have to explain that one, since it doesn't make sense as written. > > > The context switches and the pain to handle the switches makes > > handling of the requests in lesser threads which is scalable. > > Nor does that. What are you categorizing as a "lesser thread"? What makes > those any more scalable than any other kind of thread? You're free to set > the thread pool limit to any value you want, including infinity. Setting > the limit to larger than what the logical and physical resources of the > system you're running on can handle will induce performance problems (e.g., > page thrashing, excessive GCs) or outright failures as soon as someone > decides to start pounding on your server. > > - Chuck > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is thus for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all computers. > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
RE: Tomcat threads
"scalable" also seems to be a relative term here, and there are well documented strategies for scalability. So, the question is, are you just looking for strategies for scalability or do you have a real problem with load? -Original Message- From: Bharath Vasudevan [mailto:bharath@gmail.com] Sent: Tuesday, March 02, 2010 6:43 PM To: Tomcat Users List Subject: Re: Tomcat threads Why is it illlogical? Fast is a relative term. If the number of requests increases, the number of threads that can be handled by the system goes down . The context switches and the pain to handle the switches makes handling of the requests in lesser threads which is scalable. On Tue, Mar 2, 2010 at 4:34 PM, Caldarale, Charles R < chuck.caldar...@unisys.com> wrote: > > From: Bharath Vasudevan [mailto:bharath@gmail.com] > > Subject: Re: Tomcat threads > > > > If we get a request on a thread, let some other thread do > > the work for it and store the response info. The thread > > which does the work writes the response on that request. > > If the processing is fast, why would you go to the complexity and overhead > of switching to another thread to process the request? > > You should also read the servlet spec, in particular SRV.2.3.3.3: > > "Implementations of the request and response objects are not guaranteed to > be thread > safe. This means that they should only be used within the scope of the > request handling > thread. > > "References to the request and response objects should not be given to > objects > executing in other threads as the resulting behavior may be > nondeterministic. If > the thread created by the application uses the container-managed objects, > such as > the request or response object, those objects must be accessed only within > the > servlet's service life cycle and such thread itself should have a life > cycle within > the life cycle of the servlet's service method because accessing those > objects > after the service method ends may cause undeterministic problems." > > The illogical behavior you're asking for simply isn't allowed. > > - Chuck > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is thus for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all computers. > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat threads
> From: Bill Barker [mailto:billwbar...@verizon.net] > Subject: Re: Tomcat threads > > The examples in Tomcat 7 are mostly server-push (e.g. fake stock > ticker, chat). These are done by not completing the connection, > and just pumping more data to a chunked response (so you get the > spinning wheel the entire time the apps is running). An interesting trick; thanks for the info. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat threads
"Caldarale, Charles R" wrote in message news:99c8b2929b39c24493377ac7a121e21f96cb817...@usea-exch8.na.uis.unisys.com... From: Bill Barker [mailto:billwbar...@verizon.net] Subject: Re: Tomcat threads basically the request is handed off to another thread and the original one returns to the pool. This also allows for two-way communication instead of relying on polling. What's required on the client end? You can't simply push HTML or other content to a browser. Is a specialized (or at least scripted) client required? Mostly useful for custom clients channeling under HTTP, but you could probably write a scripted client as well that works with it (haven't tried). The examples in Tomcat 7 are mostly server-push (e.g. fake stock ticker, chat). These are done by not completing the connection, and just pumping more data to a chunked response (so you get the spinning wheel the entire time the apps is running). They work in most browsers (only most, because I've only tested them in IE and FireFox). But as I said before, would probably help the OP, assuming that traffic isn't too great. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat threads
> From: Bill Barker [mailto:billwbar...@verizon.net] > Subject: Re: Tomcat threads > > basically the request is handed off to another thread and the > original one returns to the pool. This also allows for two-way > communication instead of relying on polling. What's required on the client end? You can't simply push HTML or other content to a browser. Is a specialized (or at least scripted) client required? - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat threads
> From: Bharath Vasudevan [mailto:bharath@gmail.com] > Subject: Re: Tomcat threads > > Why is it illlogical? 40+ years of system architecture experience. > If the number of requests increases, the number of threads > that can be handled by the system goes down. You'll have to explain that one, since it doesn't make sense as written. > The context switches and the pain to handle the switches makes > handling of the requests in lesser threads which is scalable. Nor does that. What are you categorizing as a "lesser thread"? What makes those any more scalable than any other kind of thread? You're free to set the thread pool limit to any value you want, including infinity. Setting the limit to larger than what the logical and physical resources of the system you're running on can handle will induce performance problems (e.g., page thrashing, excessive GCs) or outright failures as soon as someone decides to start pounding on your server. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat threads
"Caldarale, Charles R" wrote in message news:99c8b2929b39c24493377ac7a121e21f96cb817...@usea-exch8.na.uis.unisys.com... From: Bharath Vasudevan [mailto:bharath@gmail.com] Subject: Re: Tomcat threads If we get a request on a thread, let some other thread do the work for it and store the response info. The thread which does the work writes the response on that request. If the processing is fast, why would you go to the complexity and overhead of switching to another thread to process the request? You should also read the servlet spec, in particular SRV.2.3.3.3: "Implementations of the request and response objects are not guaranteed to be thread safe. This means that they should only be used within the scope of the request handling thread. "References to the request and response objects should not be given to objects executing in other threads as the resulting behavior may be nondeterministic. If the thread created by the application uses the container-managed objects, such as the request or response object, those objects must be accessed only within the servlet's service life cycle and such thread itself should have a life cycle within the life cycle of the servlet's service method because accessing those objects after the service method ends may cause undeterministic problems." The illogical behavior you're asking for simply isn't allowed. At least not until Tomcat 7 ;). In there, you can do exactly what the user is asking for using the AsyncContext. The details are in the Servlet 3.0 spec, but basically the request is handed off to another thread and the original one returns to the pool. This also allows for two-way communication instead of relying on polling. Of course Comet offers similar functionality in Tomcat 6 if you don't mind being bound to Tomcat. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat threads
Why is it illlogical? Fast is a relative term. If the number of requests increases, the number of threads that can be handled by the system goes down . The context switches and the pain to handle the switches makes handling of the requests in lesser threads which is scalable. On Tue, Mar 2, 2010 at 4:34 PM, Caldarale, Charles R < chuck.caldar...@unisys.com> wrote: > > From: Bharath Vasudevan [mailto:bharath@gmail.com] > > Subject: Re: Tomcat threads > > > > If we get a request on a thread, let some other thread do > > the work for it and store the response info. The thread > > which does the work writes the response on that request. > > If the processing is fast, why would you go to the complexity and overhead > of switching to another thread to process the request? > > You should also read the servlet spec, in particular SRV.2.3.3.3: > > "Implementations of the request and response objects are not guaranteed to > be thread > safe. This means that they should only be used within the scope of the > request handling > thread. > > "References to the request and response objects should not be given to > objects > executing in other threads as the resulting behavior may be > nondeterministic. If > the thread created by the application uses the container-managed objects, > such as > the request or response object, those objects must be accessed only within > the > servlet's service life cycle and such thread itself should have a life > cycle within > the life cycle of the servlet's service method because accessing those > objects > after the service method ends may cause undeterministic problems." > > The illogical behavior you're asking for simply isn't allowed. > > - Chuck > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is thus for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all computers. > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
RE: Tomcat threads
> From: Bharath Vasudevan [mailto:bharath@gmail.com] > Subject: Re: Tomcat threads > > If we get a request on a thread, let some other thread do > the work for it and store the response info. The thread > which does the work writes the response on that request. If the processing is fast, why would you go to the complexity and overhead of switching to another thread to process the request? You should also read the servlet spec, in particular SRV.2.3.3.3: "Implementations of the request and response objects are not guaranteed to be thread safe. This means that they should only be used within the scope of the request handling thread. "References to the request and response objects should not be given to objects executing in other threads as the resulting behavior may be nondeterministic. If the thread created by the application uses the container-managed objects, such as the request or response object, those objects must be accessed only within the servlet's service life cycle and such thread itself should have a life cycle within the life cycle of the servlet's service method because accessing those objects after the service method ends may cause undeterministic problems." The illogical behavior you're asking for simply isn't allowed. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat threads
Thanks Chuck. One thing I am not able to get is why would the user care if the response was done from a different thread? Is it not going to be opaque to the user as long as its fast? If we get a request on a thread, let some other thread do the work for it and store the response info. The thread which does the work writes the response on that request. Regards, Bharath On Tue, Mar 2, 2010 at 3:57 PM, Caldarale, Charles R < chuck.caldar...@unisys.com> wrote: > > From: Bharath Vasudevan [mailto:bharath@gmail.com] > > Subject: Tomcat threads > > > > Are they user level threads or kernel threads? > > Depends on the particular JVM being used, not on Tomcat. Pretty much all > modern JVMs use a kernel thread for each java.lang.Thread instance on which > start() has been called. > > > I see that the requests are kicked off through the threads from a pool > > (which has a configured size), how can this be scalable? > > Because you can make the limit as large as you want. Making it too large > leaves you open to DOS attacks and running out of resources on the server. > > > Also, from my understanding in servlets, the response for a > > request in has to be done then and there (as in I cant respond > > for a request later in future from some random thread) . > > That's the nature of HTTP, not servlets in particular. A properly designed > application facing a long-duration situation will usually send an interim > status back to the client, and kick off an auxiliary thread to do whatever > is going to run for a while. The auxiliary thread places the final response > in some location associated with the client when it's ready. The client > polls for status and the final response using JavaScript or some other > mechanism. > > > If there is contention for a resource which might take time > > (causing the response to take time), wont tomcat run out of > > threads? > > Yes; Tomcat won't care, but your users might. Don't use poorly designed > applications. > > - Chuck > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is thus for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all computers. > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
RE: Tomcat threads
> From: Bharath Vasudevan [mailto:bharath@gmail.com] > Subject: Tomcat threads > > Are they user level threads or kernel threads? Depends on the particular JVM being used, not on Tomcat. Pretty much all modern JVMs use a kernel thread for each java.lang.Thread instance on which start() has been called. > I see that the requests are kicked off through the threads from a pool > (which has a configured size), how can this be scalable? Because you can make the limit as large as you want. Making it too large leaves you open to DOS attacks and running out of resources on the server. > Also, from my understanding in servlets, the response for a > request in has to be done then and there (as in I cant respond > for a request later in future from some random thread) . That's the nature of HTTP, not servlets in particular. A properly designed application facing a long-duration situation will usually send an interim status back to the client, and kick off an auxiliary thread to do whatever is going to run for a while. The auxiliary thread places the final response in some location associated with the client when it's ready. The client polls for status and the final response using JavaScript or some other mechanism. > If there is contention for a resource which might take time > (causing the response to take time), wont tomcat run out of > threads? Yes; Tomcat won't care, but your users might. Don't use poorly designed applications. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Tomcat threads
Hi, I would like to know more on the threads created in tomcat. Are they user level threads or kernel threads? I see that the requests are kicked off through the threads from a pool (which has a configured size), how can this be scalable? Also, from my understanding in servlets, the response for a request in has to be done then and there (as in I cant respond for a request later in future from some random thread) . If there is contention for a resource which might take time (causing the response to take time), wont tomcat run out of threads? Regards, Bharath
Re: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity
Pid wrote: I am not a number! That betrays your age too, you know that ? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity
> -Original Message- > From: Rainer Jung [mailto:rainer.j...@kippdata.de] > Sent: Friday, May 22, 2009 12:39 PM > To: Tomcat Users List > Subject: Re: Running out of tomcat threads - why many threads in > RUNNABLEstage even with no activity > > [Pantvaidya, Vishwajit] Why are connections idle for a long time not > good? I thought threads when idle take only a little memory and cpu. Are > there any other reasons? > > Because you might want to monitor connections in order to learn how many > threads you need for your load and how things grow or shrink over time. > If you keep connections open an infinite number of time, you'll only > monitor the biggest need since restart, which is often not very > interesting, because it often is artificial (triggered by some > performance slowness you might have a very big connection number created > during a short time). [Pantvaidya, Vishwajit] Good reason - I think ultimately after some immediate testing to diagnose the outofthread issues, I will use timeouts. > > d. atleast 1 server where no firewall exists has run out of threads > > Concurrency = Load * ResponseTime > > Concurrency: number of requests being processed in parallel > Load: Number of Requests per Second being handled > ResponseTime: Average Response time in seconds. > > So in case you have a performance problem and for a given load your > response time goes up by a factor of ten, the number of connections will > also go up by a factpr of 10. That's most often the reason for d) and > was the reason, why we asked for thread dumps. [Pantvaidya, Vishwajit] Again good explanation and makes a lot of sense - I do seem to remember we had performance problems on that machine. Will keep this in mind and monitor threads and take dumps if outofthreads reoccurs on that server. > > - Bring workers.properties settings in line with Apache recommendations: > > - Worker...cachesize=10 - set to 1 > > respectively when using Apache, remove this. Rely on the defaults for > that one. [Pantvaidya, Vishwajit] Sure will do - once we migrate to jk 1.2.28. > > > - Worker...cache_timeout=600 - remove > > - Worker...recycle_timeout=300 - remove > > Hmmm. [Pantvaidya, Vishwajit] Considering the excellent reasons you have given above - ultimately I will retain timeouts. But for testing firewall issues, I need to rollback connTimeout in server.xml and to make sure that my settings are consistent I need to rollback the above timeouts also. Again thanks - I think I have reasonable explanations for most of the issues / conflicting observations. This thread may be quiet for some time as I do more testing as per the actions I mentioned - will get back with results and final conclusions later. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity
On 22.05.2009 21:09, Pantvaidya, Vishwajit wrote: >> -Original Message- >> From: Rainer Jung [mailto:rainer.j...@kippdata.de] >> Sent: Friday, May 22, 2009 2:53 AM >> To: Tomcat Users List >> Subject: Re: Running out of tomcat threads - why many threads in >> RUNNABLEstage even with no activity >> >> My point is: persistent connections are good, but connections which are >> idle for a long time are not as good, so close them after some idle >> time, like e.g. 10 minutes. Of course this means you need to create new >> ones once your load goes up again, but that's not a big problem. > > [Pantvaidya, Vishwajit] Why are connections idle for a long time not good? I > thought threads when idle take only a little memory and cpu. Are there any > other reasons? Because you might want to monitor connections in order to learn how many threads you need for your load and how things grow or shrink over time. If you keep connections open an infinite number of time, you'll only monitor the biggest need since restart, which is often not very interesting, because it often is artificial (triggered by some performance slowness you might have a very big connection number created during a short time). Second: because they are making trouble so often in combination with firewalls. So in general I like persistent connections as long as they are closed when idle for a longer time. So usually I set the pool min size to 0 and the idle connection timeout to 10 minutes. > Thanks a lot Rainer, Chuck, Chris, Andre, Pid, Martin and everyone else I > missed. I spent quite some time yesterday chewing on everything I gathered in > the last few days' interactions and the conflicting behavior we are seeing in > our systems - that led to following conclusions and action plan: > > Behavior observed in diff production systems: > a. medium-to-large thread count whether firewall exists or not > b. % of runnable threads is much higher where firewall between httpd/tomcat > c. atleast 1 server where firewall exists has run out of threads > d. atleast 1 server where no firewall exists has run out of threads Concurrency = Load * ResponseTime Concurrency: number of requests being processed in parallel Load: Number of Requests per Second being handled ResponseTime: Average Response time in seconds. So in case you have a performance problem and for a given load your response time goes up by a factor of ten, the number of connections will also go up by a factpr of 10. That's most often the reason for d) and was the reason, why we asked for thread dumps. > Conclusions: > 1. In general, runnable threads should not be a prob, unless they correspond > to dropped connections. Since on our servers that have firewall between httpd > and tomcat, runnable connections are not being used for new requests and > tomcat keeps on creating new threads (leading to #b/c above), those threads > could correspond to: > i. connections dropped by firewall or > ii. hanging tomcat threads as httpd recycle timeout disconnected the > connection from that side (and there was no connectiontimeout in server.xml > so that tomcat could do the same) or > iii. combination of these "i" and "ii" > 2. Runnable threads on servers where no firewall exist (and we do not see > server running out of threads) should not be a point of concerns as they do > not correspond to dropped connections, as seen from netstat o/p at the end of > this email. So #a above could be ignored. > 3. Observation #d above is puzzling and currently I have no answers for that If d) happens again, do some thread dumps. > Action: > - check both sides by using "netstat -anop" (Apache side and the Tomcat side > without connectionTimeout, so you can see the problem in the original form). > See whether the number of AJP connections in the various TCP states differs > much between the netstat output on the Apache and on the Tomcat system. > - Bring workers.properties settings in line with Apache recommendations: > - Worker...cachesize=10 - set to 1 respectively when using Apache, remove this. Rely on the defaults for that one. > - Worker...cache_timeout=600 - remove > - Worker...recycle_timeout=300 - remove Hmmm. > Netstat o/p's: connector running on 21005, no firewall between httpd/tomcat > > Httpd Side: > > Proto Recv-Q Send-Q Local Address Foreign Address > State PID/Program nameTimer > tcp0 0 129.41.29.241:53777 129.41.29.48:21005 > ESTABLISHED - keepalive (2869.65/0/0) > tcp0 0 129.41.29.241:53943 129.41.29.48:21005 > ESTABLISHED - keepalive (3341
RE: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity
> From: Pantvaidya, Vishwajit [mailto:vpant...@selectica.com] > Subject: RE: Running out of tomcat threads - why many threads in > RUNNABLEstage even with no activity > > [Pantvaidya, Vishwajit] Why are connections idle for a > long time not good? He didn't say they weren't good, he said they weren't _as_ good - a relative, not absolute, assessment. Things sitting around idle are not /as good/ as things in use. It doesn't make them bad. > 1. In general, runnable threads should not be a prob, unless they > correspond to dropped connections. Correct; need to focus on why the connections are becoming unusable. > 3. Observation #d above is puzzling and currently I have > no answers for that That is strange; is there anything else between httpd and Tomcat in that situation? (E.g., an overly clever router?) > Netstat o/p's: connector running on 21005, no firewall between > httpd/tomcat That all looks reasonable to me - the port numbers match. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity
> -Original Message- > From: Rainer Jung [mailto:rainer.j...@kippdata.de] > Sent: Friday, May 22, 2009 2:53 AM > To: Tomcat Users List > Subject: Re: Running out of tomcat threads - why many threads in > RUNNABLEstage even with no activity > > My point is: persistent connections are good, but connections which are > idle for a long time are not as good, so close them after some idle > time, like e.g. 10 minutes. Of course this means you need to create new > ones once your load goes up again, but that's not a big problem. [Pantvaidya, Vishwajit] Why are connections idle for a long time not good? I thought threads when idle take only a little memory and cpu. Are there any other reasons? Thanks a lot Rainer, Chuck, Chris, Andre, Pid, Martin and everyone else I missed. I spent quite some time yesterday chewing on everything I gathered in the last few days' interactions and the conflicting behavior we are seeing in our systems - that led to following conclusions and action plan: Behavior observed in diff production systems: a. medium-to-large thread count whether firewall exists or not b. % of runnable threads is much higher where firewall between httpd/tomcat c. atleast 1 server where firewall exists has run out of threads d. atleast 1 server where no firewall exists has run out of threads Conclusions: 1. In general, runnable threads should not be a prob, unless they correspond to dropped connections. Since on our servers that have firewall between httpd and tomcat, runnable connections are not being used for new requests and tomcat keeps on creating new threads (leading to #b/c above), those threads could correspond to: i. connections dropped by firewall or ii. hanging tomcat threads as httpd recycle timeout disconnected the connection from that side (and there was no connectiontimeout in server.xml so that tomcat could do the same) or iii. combination of these "i" and "ii" 2. Runnable threads on servers where no firewall exist (and we do not see server running out of threads) should not be a point of concerns as they do not correspond to dropped connections, as seen from netstat o/p at the end of this email. So #a above could be ignored. 3. Observation #d above is puzzling and currently I have no answers for that Action: - check both sides by using "netstat -anop" (Apache side and the Tomcat side without connectionTimeout, so you can see the problem in the original form). See whether the number of AJP connections in the various TCP states differs much between the netstat output on the Apache and on the Tomcat system. - Bring workers.properties settings in line with Apache recommendations: - Worker...cachesize=10 - set to 1 - Worker...cache_timeout=600 - remove - Worker...recycle_timeout=300 - remove Netstat o/p's: connector running on 21005, no firewall between httpd/tomcat Httpd Side: Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program nameTimer tcp0 0 129.41.29.241:53777 129.41.29.48:21005 ESTABLISHED - keepalive (2869.65/0/0) tcp0 0 129.41.29.241:53943 129.41.29.48:21005 ESTABLISHED - keepalive (3341.39/0/0) tcp0 0 129.41.29.241:49950 129.41.29.48:21005 ESTABLISHED - keepalive (6701.51/0/0) tcp0 0 129.41.29.241:49927 129.41.29.48:21005 ESTABLISHED - keepalive (6240.25/0/0) tcp0 0 129.41.29.241:49926 129.41.29.48:21005 ESTABLISHED - keepalive (6239.47/0/0) tcp0 0 129.41.29.241:49971 129.41.29.48:21005 ESTABLISHED - keepalive (6931.40/0/0) tcp0 0 129.41.29.241:49868 129.41.29.48:21005 ESTABLISHED - keepalive (5743.83/0/0) tcp0 0 129.41.29.241:49865 129.41.29.48:21005 ESTABLISHED - keepalive (5741.65/0/0) tcp0 0 129.41.29.241:49867 129.41.29.48:21005 ESTABLISHED - keepalive (5743.16/0/0) tcp0 0 129.41.29.241:49901 129.41.29.48:21005 ESTABLISHED - keepalive (5906.92/0/0) tcp0 0 129.41.29.241:49795 129.41.29.48:21005 ESTABLISHED - keepalive (4659.11/0/0) tcp0 0 129.41.29.241:49558 129.41.29.48:21005 ESTABLISHED - keepalive (1705.06/0/0) tcp0 0 129.41.29.241:50796 129.41.29.48:21005 ESTABLISHED - keepalive (4551.79/0/0) tcp0 0 129.41.29.241:50784 129.41.29.48:21005 ESTABLISHED - keepalive (4539.53/0/0) tcp
RE: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity
> From: Chetan Chheda [mailto:chetan_chh...@yahoo.com] > Subject: Re: Running out of tomcat threads - why many threads in > RUNNABLEstage even with no activity > As with Vishwajit, my tomcat ends up with all threads busy and not > serving any new requests. After setting the connectionTimeout the > threads are being recycled but apache is not liking .. as per the > messsage in mod_jk.log Again, the most likely cause is something between httpd and Tomcat that is silently dropping the connections; a badly behaving firewall is a possible culprit. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity
Chris, As with Vishwajit, my tomcat ends up with all threads busy and not serving any new requests. After setting the connectionTimeout the threads are being recycled but apache is not liking .. as per the messsage in mod_jk.log Chetan From: Christopher Schultz To: Tomcat Users List Sent: Friday, May 22, 2009 8:37:52 AM Subject: Re: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Chetan, On 5/21/2009 2:08 PM, Chetan Chheda wrote: > I am following this thread with great interest. I have a similar > issue as Vishwajit and have resorted to adding the connectionTimeout > to get rid of a large number of RUNNABLE threads. Why? Are you just offended by the number of threads, or do you have a legitimate resource problem? > But mod_jk does not like tomcat threads timing out and logs the > message "increase the backend idle connection timeout or the > connection_pool_minsize" in the mod_jk logs > > which leads me to believe that its apache thats not letting go of the > threads in my case. Again, you need to set the Connector's connectionTimeout /and/ your workers' connection_pool_timeout settings to the same time interval (note that they use different semantics... one is in seconds and the other is in ms, so read the documentation carefully). - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkoWnKAACgkQ9CaO5/Lv0PBcTwCbBhuJ8/nwYLq/LAxCSVDer35t jAIAn2oUL3on6ki/x9pZHn8n0tLuVS8H =y10X -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Chetan, On 5/21/2009 2:08 PM, Chetan Chheda wrote: > I am following this thread with great interest. I have a similar > issue as Vishwajit and have resorted to adding the connectionTimeout > to get rid of a large number of RUNNABLE threads. Why? Are you just offended by the number of threads, or do you have a legitimate resource problem? > But mod_jk does not like tomcat threads timing out and logs the > message "increase the backend idle connection timeout or the > connection_pool_minsize" in the mod_jk logs > > which leads me to believe that its apache thats not letting go of the > threads in my case. Again, you need to set the Connector's connectionTimeout /and/ your workers' connection_pool_timeout settings to the same time interval (note that they use different semantics... one is in seconds and the other is in ms, so read the documentation carefully). - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkoWnKAACgkQ9CaO5/Lv0PBcTwCbBhuJ8/nwYLq/LAxCSVDer35t jAIAn2oUL3on6ki/x9pZHn8n0tLuVS8H =y10X -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity
On 22.05.2009 03:54, Pantvaidya, Vishwajit wrote: >> -Original Message- >> From: Rainer Jung [mailto:rainer.j...@kippdata.de] >> Sent: Thursday, May 21, 2009 3:37 PM >> To: Tomcat Users List >> Subject: Re: Running out of tomcat threads - why many threads in >> RUNNABLEstage even with no activity >> >> On 22.05.2009 00:19, Pantvaidya, Vishwajit wrote: >>> [Pantvaidya, Vishwajit] I will set >>> - cachesize=1 (doc says jk will autoset this value only for worker-mpm >> and we use httpd 2.0 prefork) >> >> You don't have to: "JK will discover this number for the Apache web >> server automatically and set the pool size to this value". > > [Pantvaidya, Vishwajit] Does what you say hold true for jk 1.2.15 also? > Because I saw that for the 1.2.15 cachesize directive, > http://tomcat.apache.org/connectors-doc/reference/workers.html#Deprecated%20Worker%20Directives > says that "JK will discover the number of threads per child process on > Apache 2 web server with worker-mpm and set its default value to match the > ThreadsPerChild Apache directive.". Since we use pre-fork MPM, I assumed we > need to set cachesize. I would say yes, but now your Ops people who resist upgrading try to play my time against their time. I'm not going to look it up for them, they should upgrade ;) >>> - remove cache and recycle timeouts >> Chris and me are not having the same opinion here. You can choose :) >> > [Pantvaidya, Vishwajit] I think that may be only because my adding the > connectionTimeout led you to believe that I wanted nonpersistent conn's. Now > that I know persistent connections are better, I am trying to rollback > connectionTimeout - and then I guess you will agree with Chris that I need to > rollback the recycletimeouts, etc in workers file on httpd side also? My point is: persistent connections are good, but connections which are idle for a long time are not as good, so close them after some idle time, like e.g. 10 minutes. Of course this means you need to create new ones once your load goes up again, but that's not a big problem. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity
> -Original Message- > From: Rainer Jung [mailto:rainer.j...@kippdata.de] > Sent: Thursday, May 21, 2009 3:37 PM > To: Tomcat Users List > Subject: Re: Running out of tomcat threads - why many threads in > RUNNABLEstage even with no activity > > On 22.05.2009 00:19, Pantvaidya, Vishwajit wrote: > > [Pantvaidya, Vishwajit] I will set > > - cachesize=1 (doc says jk will autoset this value only for worker-mpm > and we use httpd 2.0 prefork) > > You don't have to: "JK will discover this number for the Apache web > server automatically and set the pool size to this value". [Pantvaidya, Vishwajit] Does what you say hold true for jk 1.2.15 also? Because I saw that for the 1.2.15 cachesize directive, http://tomcat.apache.org/connectors-doc/reference/workers.html#Deprecated%20Worker%20Directives says that "JK will discover the number of threads per child process on Apache 2 web server with worker-mpm and set its default value to match the ThreadsPerChild Apache directive.". Since we use pre-fork MPM, I assumed we need to set cachesize. > > - remove cache and recycle timeouts > > Chris and me are not having the same opinion here. You can choose :) > [Pantvaidya, Vishwajit] I think that may be only because my adding the connectionTimeout led you to believe that I wanted nonpersistent conn's. Now that I know persistent connections are better, I am trying to rollback connectionTimeout - and then I guess you will agree with Chris that I need to rollback the recycletimeouts, etc in workers file on httpd side also? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity
On 22.05.2009 00:19, Pantvaidya, Vishwajit wrote: > [Pantvaidya, Vishwajit] I will set > - cachesize=1 (doc says jk will autoset this value only for worker-mpm and we > use httpd 2.0 prefork) You don't have to: "JK will discover this number for the Apache web server automatically and set the pool size to this value". > - remove cache and recycle timeouts Chris and me are not having the same opinion here. You can choose :) > But before all this, I will retest after removing connectionTimeout in > server.xml - just to test if there are firewall caused issues as mentioned > above. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity
> -Original Message- > From: Christopher Schultz [mailto:ch...@christopherschultz.net] > Sent: Thursday, May 21, 2009 10:05 AM > To: Tomcat Users List > Subject: Re: Running out of tomcat threads - why many threads in > RUNNABLEstage even with no activity > > > Vishwajit, > > On 5/20/2009 3:01 PM, Pantvaidya, Vishwajit wrote: > > [Pantvaidya, Vishwajit] Ok so RUNNABLE i.e. persistent threads should > > not be an issue. The only reason why I thought that was an issue was > > that I was observing that the none of the RUNNABLE connections were > > not being used to serve new requests, only the WAITING ones were - > > and I do know for sure that the RUNNABLE threads were not servicing > > any existing requests as I was the only one using the system then. > > It seems pretty clear that this is what your problem is. See if you can > follow the order of events described below: > > 1. Tomcat and Apache httpd are started. httpd makes one or more >(persistent) AJP connections to Tomcat and holds them open (duh). >Each connection from httpd->Tomcat puts a Java thread in the RUNNABLE >state (though actually blocked on socket read, it's not "really" >runnable) > > 2. Some requests are received by httpd and sent over the AJP connections >to Tomcat (or not ... it really doesn't matter) > > 3. Time passes, your recycle_timeout (300s) or cache_timeout (600s) >expires > > 4. A new request comes in to httpd destined for Tomcat. mod_jk dutifully >follows your instructions for closing the connections expired in #3 >above (note that Tomcat has no idea that the connection has been >closed, and so those threads remain in the RUNNABLE state, not >connected to anything, lost forever) > > 5. A new connection (or multiple new connections... not sure exactly >how mod_jks connection expiration-and-reconnect logic is done) >is made to Tomcat which allocates a new thread (or threads) >which is/are in the RUNNABLE state > > Rinse, repeat, your server chokes to death when it runs out of threads. > > The above description accounts for your "loss" of 4 threads at a time: > your web browser requests the initial page followed by 3 other assets > (image, css, whatever). Each one of them hits step #4 above, causing a > new AJP connection to be created, with the old one still hanging around > on the Tomcat side just wasting a thread and memory. > > By setting connectionTimeout on the AJP , you are /doing what > you should have done in the first place, which is match mod_jk > cache_timeout with Connector connectionTimeout/. This allows the threads > on the Tomcat side to expire just like those on the httpd side. They > should expire at (virtually) the same time and everything works as > expected. [Pantvaidya, Vishwajit] Thanks Chris - all this makes a lot of sense. However I am not seeing same problem (tomcat running out of threads) on other servers which are running exactly same configuration except that in those cases is no firewall separating websvr and tomcat. Here are the figures of RUNNABLE on 3 different tomcat server running same config: 1. Firewall between httpd and tomcat - 120 threads, 112 runnable (93%) 2. No firewall between httpd and tomcat - 40 threads, 11 runnable (27%) 3. No firewall between httpd and tomcat - 48 threads, 2 runnable (4%) Leads me to believe there is some firewall related mischief happening with #1. > This problem is compounded by your initial configuration which created > 10 connections from httpd->Tomcat for every (prefork) httpd process, > resulting in 9 useless AJP connections for every httpd process. I > suspect that you were expiring 10 connections at a time instead of just > one, meaning that you were running out of threads 10 times faster than > you otherwise would. [Pantvaidya, Vishwajit] I did not note connections expiring in multiple of 10. But I will keep an eye out for this. However from the cachesize explanation at http://tomcat.apache.org/connectors-doc/reference/workers.html#Deprecated%20Worker%20Directives I get the impression that this value imposes an upper limit - meaning it may not necessarily create 10 tomcat/jk connections for an httpd child process > Suggestions: > 1. Tell your ops guys we know what we're talking about > 2. Upgrade mod_jk > 3. Set connection_pool_size=1, or, better yet, remove the config >altogether and let mod_jk determine its own value > 4. Remove all timeouts unless you know that you have a misbehaving >firewall. If you do, enable cping/cpong (the preferred strategy >by at least one author or mod_jk) > > - -chris [Pantvaidya, Vishwajit] I will set - cachesize=
Re: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity
On 21.05.2009 20:59, Pantvaidya, Vishwajit wrote: >> 3) I think I already indicated that you do not want to look at entries >> in TIME_WAIT state. This state is special and not related to any threads > > [Pantvaidya, Vishwajit] My netstat o/p had FIN_WAIT and CLOSE_WAIT, but not > TIMED_WAIT. Did some reading on TCP states, and seems to me that next time I > do a netstat (w/o the connTimeout in server.xml), I can ignore processes with > all these wait states as they seem to just indicate connections in different > stages of closing. No, you can not ignore CLOSE_WAIT, FIN_WAIT, FIN_WAIT2 and maybe others. >> 4) Firewall idle connection drop: First read >> >> http://tomcat.apache.org/connectors- >> doc/generic_howto/timeouts.html#Firewall%20Connection%20Dropping >> >> carefully and try to understand. >> >> Any mod_jk attribute that takes a booelan value will accept 1, true, >> True, t or T as true, 0, false, False, f or F as false (and maybe even >> more). > > [Pantvaidya, Vishwajit] Our workers.props file has most recommended settings: > Worker...type=ajp13 > Worker...cachesize=10 > Worker...cache_timeout=600 > Worker...socket_keepalive=1 > Worker...recycle_timeout=300 It doesn't make sense for me checking values for attributes that do no longer exist since a long time. > We are not setting connectionpoolsize and minsize - but from the timeouts doc > that should be okay as JK auto adjusts to httpd poolsize. So I think only > thing left is to remove connectionTimeout and test. No. First cachesize is equivalent to connection pool size. If you want to get all idle connections closed you will also need the min pool size. Also important are the cping/cpong features. All that applies to recent versions. > Your link recommends connectionTimeouts and JKoption +DisableReuse as a final > option - I think that will remove persistent connections on httpd and tomcat > side. For us, the connectionTimeout alone worked. And my netstat o/p showing > 11 conns on http side and only 2 on tomcat side means our http conn's are > persistent while tomcat ones are not, right?. So I am thinking the perf > downside should be better than if I had set +DisableReuse also? You would only use DisableReuse if the other configuration options do not help. Usually they do help, so you won't use DisableReuse. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity
> > 1) If you want to analyze your original problem, you need to get back to > the original situation, i.e. without connectionTimeout. It doesn't make > much sense to guess about the original problem by looking at something > very different. [Pantvaidya, Vishwajit] Yes - I have already initiated that. Expect to be able to test the system without connectionTimeout tonight. > 2) The output of netstat and the content of a thread dump change in > time. If you want to understand the exact relations between the two > netstat outputs and a thread dump, you need to ensure to produce those > three things as close in time as possible. I'm talking about seconds not > milliseconds. [Pantvaidya, Vishwajit] Yes, I had taken the netstats and the threads dumps pretty close together. But I will try and append a timestamp next time I take the output. > 3) I think I already indicated that you do not want to look at entries > in TIME_WAIT state. This state is special and not related to any threads [Pantvaidya, Vishwajit] My netstat o/p had FIN_WAIT and CLOSE_WAIT, but not TIMED_WAIT. Did some reading on TCP states, and seems to me that next time I do a netstat (w/o the connTimeout in server.xml), I can ignore processes with all these wait states as they seem to just indicate connections in different stages of closing. > 4) Firewall idle connection drop: First read > > http://tomcat.apache.org/connectors- > doc/generic_howto/timeouts.html#Firewall%20Connection%20Dropping > > carefully and try to understand. > > Any mod_jk attribute that takes a booelan value will accept 1, true, > True, t or T as true, 0, false, False, f or F as false (and maybe even > more). [Pantvaidya, Vishwajit] Our workers.props file has most recommended settings: Worker...type=ajp13 Worker...cachesize=10 Worker...cache_timeout=600 Worker...socket_keepalive=1 Worker...recycle_timeout=300 We are not setting connectionpoolsize and minsize - but from the timeouts doc that should be okay as JK auto adjusts to httpd poolsize. So I think only thing left is to remove connectionTimeout and test. Your link recommends connectionTimeouts and JKoption +DisableReuse as a final option - I think that will remove persistent connections on httpd and tomcat side. For us, the connectionTimeout alone worked. And my netstat o/p showing 11 conns on http side and only 2 on tomcat side means our http conn's are persistent while tomcat ones are not, right?. So I am thinking the perf downside should be better than if I had set +DisableReuse also? > 5) Matching port numbers > > Usually the port numbers should match. The non matching of the port > numbers could indicate, that there is a firewall in between, although > most firewall systems will be transparent to the ports (yes, I know > there are many variations). Since the port numbers are very close I > would guess, that the reason for not matching is that netstat was done a > couple of seconds or more apart, and your connections are only used for > a very short time, so we have lots of new connections activity. [Pantvaidya, Vishwajit] Yes - I confirmed from admins that there is a firewall and I will work with them to understand that side more. Our connection timeout are in the order of 10 mins - so I am not sure why the port#s don't match - will try and find if the firewall is having different port# ranges configured for httpd & tomcat side. > 6) TCP states > > LISTEN on the Tomcat side corresponds to the one TP thread, that does a > socket accept. > > ESTABLISHED: both sides still want to use this connection. On the Tomcat > side shows up as socketRead0() > > CLOSE_WAIT: the other side has closed the connection, the local side > hasn't yet. E.g. if Tomcat closes the connection because of > connectionTimeout, but Apache doesn't have a configured idle timeout and > didn't yet try to reuse the half-closed connection, the connection will > be shown as CLOSE_WAIT on the httpd side. If Apache closed the > connection, but Tomcat hasn't noticed yet, it will be CLOSE_WAIT at the > Tomcat end. In this case it could be also socketRead0() in the thread > dump. > > FIN_WAIT2: most likely the other end of CLOSE_WAIT. [Pantvaidya, Vishwajit] This is interesting because all the conn's in the netstat o/p on httpd and tomcat sides which involved the connector port 21065 (either in local/foreign addr) were in WAIT states. But I was seeing one RUNNABLE thread in socketAccept in the thread console. But anyway I will redo this whole thing with the conntimeouts removed and make sure I take the netstats and thread dumps in close succession. > 7) mod_jk update > > Before you start to fix your mod_jk configuration, go to your ops people > and tell them that they are using a very bad mod_jk version and they > have to update. The right version to update to is 1.2.28. It does make > no sense at all to try to fix this with your old version. Solve your > problem in the right way, by setting much more attr
Re: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity
I am following this thread with great interest. I have a similar issue as Vishwajit and have resorted to adding the connectionTimeout to get rid of a large number of RUNNABLE threads. But mod_jk does not like tomcat threads timing out and logs the message "increase the backend idle connection timeout or the connection_pool_minsize" in the mod_jk logs which leads me to believe that its apache thats not letting go of the threads in my case. From: Christopher Schultz To: Tomcat Users List Sent: Thursday, May 21, 2009 1:05:21 PM Subject: Re: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Vishwajit, On 5/20/2009 3:01 PM, Pantvaidya, Vishwajit wrote: > [Pantvaidya, Vishwajit] Ok so RUNNABLE i.e. persistent threads should > not be an issue. The only reason why I thought that was an issue was > that I was observing that the none of the RUNNABLE connections were > not being used to serve new requests, only the WAITING ones were - > and I do know for sure that the RUNNABLE threads were not servicing > any existing requests as I was the only one using the system then. It seems pretty clear that this is what your problem is. See if you can follow the order of events described below: 1. Tomcat and Apache httpd are started. httpd makes one or more (persistent) AJP connections to Tomcat and holds them open (duh). Each connection from httpd->Tomcat puts a Java thread in the RUNNABLE state (though actually blocked on socket read, it's not "really" runnable) 2. Some requests are received by httpd and sent over the AJP connections to Tomcat (or not ... it really doesn't matter) 3. Time passes, your recycle_timeout (300s) or cache_timeout (600s) expires 4. A new request comes in to httpd destined for Tomcat. mod_jk dutifully follows your instructions for closing the connections expired in #3 above (note that Tomcat has no idea that the connection has been closed, and so those threads remain in the RUNNABLE state, not connected to anything, lost forever) 5. A new connection (or multiple new connections... not sure exactly how mod_jks connection expiration-and-reconnect logic is done) is made to Tomcat which allocates a new thread (or threads) which is/are in the RUNNABLE state Rinse, repeat, your server chokes to death when it runs out of threads. The above description accounts for your "loss" of 4 threads at a time: your web browser requests the initial page followed by 3 other assets (image, css, whatever). Each one of them hits step #4 above, causing a new AJP connection to be created, with the old one still hanging around on the Tomcat side just wasting a thread and memory. By setting connectionTimeout on the AJP , you are /doing what you should have done in the first place, which is match mod_jk cache_timeout with Connector connectionTimeout/. This allows the threads on the Tomcat side to expire just like those on the httpd side. They should expire at (virtually) the same time and everything works as expected. This problem is compounded by your initial configuration which created 10 connections from httpd->Tomcat for every (prefork) httpd process, resulting in 9 useless AJP connections for every httpd process. I suspect that you were expiring 10 connections at a time instead of just one, meaning that you were running out of threads 10 times faster than you otherwise would. Suggestions: 1. Tell your ops guys we know what we're talking about 2. Upgrade mod_jk 3. Set connection_pool_size=1, or, better yet, remove the config altogether and let mod_jk determine its own value 4. Remove all timeouts unless you know that you have a misbehaving firewall. If you do, enable cping/cpong (the preferred strategy by at least one author or mod_jk) - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEUEARECAAYFAkoVidEACgkQ9CaO5/Lv0PCzwACYsrhskrNVgJFk6hI1gU+Kkmbe WQCfTNbSLTgNtHcTbTAu5kw5igicNMw= =0EWv -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Vishwajit, On 5/20/2009 3:01 PM, Pantvaidya, Vishwajit wrote: > [Pantvaidya, Vishwajit] Ok so RUNNABLE i.e. persistent threads should > not be an issue. The only reason why I thought that was an issue was > that I was observing that the none of the RUNNABLE connections were > not being used to serve new requests, only the WAITING ones were - > and I do know for sure that the RUNNABLE threads were not servicing > any existing requests as I was the only one using the system then. It seems pretty clear that this is what your problem is. See if you can follow the order of events described below: 1. Tomcat and Apache httpd are started. httpd makes one or more (persistent) AJP connections to Tomcat and holds them open (duh). Each connection from httpd->Tomcat puts a Java thread in the RUNNABLE state (though actually blocked on socket read, it's not "really" runnable) 2. Some requests are received by httpd and sent over the AJP connections to Tomcat (or not ... it really doesn't matter) 3. Time passes, your recycle_timeout (300s) or cache_timeout (600s) expires 4. A new request comes in to httpd destined for Tomcat. mod_jk dutifully follows your instructions for closing the connections expired in #3 above (note that Tomcat has no idea that the connection has been closed, and so those threads remain in the RUNNABLE state, not connected to anything, lost forever) 5. A new connection (or multiple new connections... not sure exactly how mod_jks connection expiration-and-reconnect logic is done) is made to Tomcat which allocates a new thread (or threads) which is/are in the RUNNABLE state Rinse, repeat, your server chokes to death when it runs out of threads. The above description accounts for your "loss" of 4 threads at a time: your web browser requests the initial page followed by 3 other assets (image, css, whatever). Each one of them hits step #4 above, causing a new AJP connection to be created, with the old one still hanging around on the Tomcat side just wasting a thread and memory. By setting connectionTimeout on the AJP , you are /doing what you should have done in the first place, which is match mod_jk cache_timeout with Connector connectionTimeout/. This allows the threads on the Tomcat side to expire just like those on the httpd side. They should expire at (virtually) the same time and everything works as expected. This problem is compounded by your initial configuration which created 10 connections from httpd->Tomcat for every (prefork) httpd process, resulting in 9 useless AJP connections for every httpd process. I suspect that you were expiring 10 connections at a time instead of just one, meaning that you were running out of threads 10 times faster than you otherwise would. Suggestions: 1. Tell your ops guys we know what we're talking about 2. Upgrade mod_jk 3. Set connection_pool_size=1, or, better yet, remove the config altogether and let mod_jk determine its own value 4. Remove all timeouts unless you know that you have a misbehaving firewall. If you do, enable cping/cpong (the preferred strategy by at least one author or mod_jk) - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEUEARECAAYFAkoVidEACgkQ9CaO5/Lv0PCzwACYsrhskrNVgJFk6hI1gU+Kkmbe WQCfTNbSLTgNtHcTbTAu5kw5igicNMw= =0EWv -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Vishwajit, On 5/19/2009 6:08 PM, Pantvaidya, Vishwajit wrote: >> No, Tomcat uses precisely 1 thread to handle each incoming HTTP >> request. If keepalives are used, multiple requests may be handled >> by the same thread before it is returned to the pool, or different >> threads may be used to serve different requests from the single >> connection, but in either case, no more than 1 thread will be used >> to service a single HTTP request. > > [Pantvaidya, Vishwajit] Could this happen if upon my http browser > request, the app could be spawning multiple redirects in quick > succession leading tomcat to create multiple threads. Any other > thoughts why I could be seeing tomcat spawn threads in multiples of > 4? No, the only thing your browser can do in order to activate a thread is to make a request. If a single page contains many items to download (images, CSS files, scripts, etc.) then the browser may make several requests to the server simultaneously. Most web browsers limit their simultaneous requests to a single server to 4, so maybe that's what you're observing. The weird thing is that you're observing 4 threads doing something /without/ any requests being made. > [Pantvaidya, Vishwajit] Ok thanks. "httpd -l" showed perfork.c. I > guess that means we are using prefork MPM. So our cachesize should be > 1? Our mod_jk version is 1.2.15 - will that also auto-detect the > cache-size? Yes and yes. > [Pantvaidya, Vishwajit] I agree - but again as I mentioned above > because the admin will be conservative about any changes, I need to > have a strong reason. How's this for a compelling reason: your server keeps locking-up in its current configuration. > Also when you say let the connection stay > alive, does that mean let the TP-Processor thread remain in Waiting > state / Runnable state? I mean that you shouldn't set any connection_pool_timeout on any of your workers. There really isn't a need to set this unless you are having some kind of problem IMO. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkoVe7oACgkQ9CaO5/Lv0PAbdQCeK088qxz+9/TNBnJgfHBg6Gyz 7+4AoI6fbXdn4Sh6kR9QaQhvK/jmHWpF =frxU -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity
Trying to add some info below. On 21.05.2009 05:09, Caldarale, Charles R wrote: >> From: Pantvaidya, Vishwajit [mailto:vpant...@selectica.com] >> Subject: RE: Running out of tomcat threads - why many threads in >> RUNNABLEstage even with no activity >> >> So socket_keepalive is already 1. So does this mean that firewall is >> dropping connections in spite of it. > > The doc does not mention using 1 here, just true (although other variables > allow either). Would be best to get Rainer's opinion when the sun comes up > in Europe. > >> My netstat o/p had only 2 tomcat connections active in FIN_WAIT2 and >> about 11 in keepalive on httpd side - I guess this does not indicate >> any hanging connections? > > It's not what one would like to see, especially since none of the ports match > - all of the connections are broken. > >> Could that be because currently connectionTimeout is >> active in my server.xml? > > I think so; it appears that setting the connectionTimeout on the Tomcat side > will effectively disable the expected persistence; it's an expensive > workaround for the problem. 1) If you want to analyze your original problem, you need to get back to the original situation, i.e. without connectionTimeout. It doesn't make much sense to guess about the original problem by looking at something very different. 2) The output of netstat and the content of a thread dump change in time. If you want to understand the exact relations between the two netstat outputs and a thread dump, you need to ensure to produce those three things as close in time as possible. I'm talking about seconds not milliseconds. 3) I think I already indicated that you do not want to look at entries in TIME_WAIT state. This state is special and not related to any threads in Apache or in Tomcat. A connection in TIME_WAIT state is in no way longer associated with a process and will no longer handle any data. It's a placeholder to prevent new connections using the same ports and possibly getting confused by old packets for the previous connections coming in late. The only reason to care about TIME_WAIT connections is, when the total number of all connections on the system (all ports and including TIME_WAIT) gets into more than 1. Some systems can cope with more, like 6, but if you go above 1, then you need to start thinking about it. I assume this is in no way the case here. So let's for the moment always forget about the TIME_WAIT connections. 4) Firewall idle connection drop: First read http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html#Firewall%20Connection%20Dropping carefully and try to understand. Any mod_jk attribute that takes a booelan value will accept 1, true, True, t or T as true, 0, false, False, f or F as false (and maybe even more). 5) Matching port numbers Usually the port numbers should match. The non matching of the port numbers could indicate, that there is a firewall in between, although most firewall systems will be transparent to the ports (yes, I know there are many variations). Since the port numbers are very close I would guess, that the reason for not matching is that netstat was done a couple of seconds or more apart, and your connections are only used for a very short time, so we have lots of new connections activity. 6) TCP states LISTEN on the Tomcat side corresponds to the one TP thread, that does a socket accept. ESTABLISHED: both sides still want to use this connection. On the Tomcat side shows up as socketRead0() CLOSE_WAIT: the other side has closed the connection, the local side hasn't yet. E.g. if Tomcat closes the connection because of connectionTimeout, but Apache doesn't have a configured idle timeout and didn't yet try to reuse the half-closed connection, the connection will be shown as CLOSE_WAIT on the httpd side. If Apache closed the connection, but Tomcat hasn't noticed yet, it will be CLOSE_WAIT at the Tomcat end. In this case it could be also socketRead0() in the thread dump. FIN_WAIT2: most likely the other end of CLOSE_WAIT. 7) mod_jk update Before you start to fix your mod_jk configuration, go to your ops people and tell them that they are using a very bad mod_jk version and they have to update. The right version to update to is 1.2.28. It does make no sense at all to try to fix this with your old version. Solve your problem in the right way, by setting much more attributes on the JK side than simply the connectionTimeout on the Tomcat side. Most important: read the above timeouts page fully. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity
> From: Pid [mailto:p...@pidster.com] > Subject: Re: Running out of tomcat threads - why many threads in > RUNNABLEstage even with no activity > > I am not a number! But are you a free man? We want information, by hook or by crook... (Apologies to the late Patrick McGoohan.) - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity
Caldarale, Charles R wrote: >> From: André Warnier [mailto:a...@ice-sa.com] >> Subject: Re: Running out of tomcat threads - why many threads in >> RUNNABLEstage even with no activity > >> if you add "-p" to netstat (at least under Linux), it will also >> show the program that corresponds to that line. > > Or at least -o to show the pid number. I am not a number! p > - Chuck > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is thus for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all computers. > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity
> From: Pantvaidya, Vishwajit [mailto:vpant...@selectica.com] > Subject: RE: Running out of tomcat threads - why many threads in > RUNNABLEstage even with no activity > > So socket_keepalive is already 1. So does this mean that firewall is > dropping connections in spite of it. The doc does not mention using 1 here, just true (although other variables allow either). Would be best to get Rainer's opinion when the sun comes up in Europe. > My netstat o/p had only 2 tomcat connections active in FIN_WAIT2 and > about 11 in keepalive on httpd side - I guess this does not indicate > any hanging connections? It's not what one would like to see, especially since none of the ports match - all of the connections are broken. > Could that be because currently connectionTimeout is > active in my server.xml? I think so; it appears that setting the connectionTimeout on the Tomcat side will effectively disable the expected persistence; it's an expensive workaround for the problem. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity
> This FAQ entry looks promising: > http://tomcat.apache.org/connectors-doc/miscellaneous/faq.html > > Look at the entry entitled "I've got a firewall between my web server and > Tomcat which drops ajp13 connections after some time". > > Configuring keep-alives is a fairly low-overhead workaround, but it would > be better to fix the firewall so it doesn't silently drop connections. > > - Chuck > [Pantvaidya, Vishwajit] Thanks Chuck. My workers.properties already has following settings: port=21065 cachesize=10 cache_timeout=600 socket_keepalive=1 recycle_timeout=300 So socket_keepalive is already 1. So does this mean that firewall is dropping connections in spite of it. About the netstat output I sent earlier - I guess an indicator of a firewall dropping connections, would be if the output showed many more active connections on the tomcat side than on the httpd side - is that accurate? My netstat o/p had only 2 tomcat connections active in FIN_WAIT2 and about 11 in keepalive on httpd side - I guess this does not indicate any hanging connections? Could that be because currently connectionTimeout is active in my server.xml? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity
> From: Martin Gainty [mailto:mgai...@hotmail.com] > Subject: RE: Running out of tomcat threads - why many threads in > RUNNABLEstage even with no activity > http://publib.boulder.ibm.com/infocenter/wasinfo/v4r0/index.jsp?topic=/ > com.ibm.support.was40.doc/html/Plug_in/swg21163659.html Vaguely interesting, but not really applicable. The protocol in question here is AJP, not HTTP, and the connections are expected to be persistent, with no keep-alives needed. Unlike in the article cited, neither end will close the connection (unless an error occurs, of course). However, when there's an ill-behaved firewall in between Tomcat and the http server (as we appear to have here), keep-alives may be required. This FAQ entry looks promising: http://tomcat.apache.org/connectors-doc/miscellaneous/faq.html Look at the entry entitled "I've got a firewall between my web server and Tomcat which drops ajp13 connections after some time". Configuring keep-alives is a fairly low-overhead workaround, but it would be better to fix the firewall so it doesn't silently drop connections. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity
definitely not TC ..the problem is with your WebServer please read this tutorial on diagnosing CLOSE_WAIT sockets from your WebServer http://publib.boulder.ibm.com/infocenter/wasinfo/v4r0/index.jsp?topic=/com.ibm.support.was40.doc/html/Plug_in/swg21163659.html Martin __ Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen. Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité pour le contenu fourni. > From: vpant...@selectica.com > To: users@tomcat.apache.org > Date: Wed, 20 May 2009 15:17:15 -0700 > Subject: RE: Running out of tomcat threads - why many threads in > RUNNABLEstage even with no activity > > > > The fact that *none* of the ports match would suggest (but not prove) > > that > > > someone in the middle is closing the connections, and not telling either > > > end about it. > > > > > > Do the netstat -anop again; it should be more interesting. > > > > > > - Chuck > > > > > > > [Pantvaidya, Vishwajit] Tomcat server port 11065, connector port 21065 > > > > On Httpd Side: > > > > Proto Recv-Q Send-Q Local Address Foreign Address > > State PID/Program nameTimer > > ... > > tcp0 0 0.0.0.0:25 0.0.0.0:* > > LISTEN - off (0.00/0/0) > > tcp1 0 129.41.29.243:44003 172.27.127.201:21065 > > CLOSE_WAIT - keepalive (7194.80/0/0) > > tcp1 0 129.41.29.243:44002 172.27.127.201:21065 > > CLOSE_WAIT - keepalive (7194.43/0/0) > > tcp1 0 129.41.29.243:44001 172.27.127.201:21065 > > CLOSE_WAIT - keepalive (7192.26/0/0) > > tcp1 0 129.41.29.243:44000 172.27.127.201:21065 > > CLOSE_WAIT - keepalive (7189.64/0/0) > > tcp1 0 129.41.29.243:43990 172.27.127.201:21065 > > CLOSE_WAIT - keepalive (7016.23/0/0) > > tcp1 0 129.41.29.243:43999 172.27.127.201:21065 > > CLOSE_WAIT - keepalive (7189.30/0/0) > > tcp1 0 129.41.29.243:43998 172.27.127.201:21065 > > CLOSE_WAIT - keepalive (7186.76/0/0) > > tcp1 0 129.41.29.243:43996 172.27.127.201:21065 > > CLOSE_WAIT - keepalive (7183.86/0/0) > > tcp1 0 129.41.29.243:43994 172.27.127.201:21065 > > CLOSE_WAIT - keepalive (7174.09/0/0) > > tcp1 0 129.41.29.243:43993 172.27.127.201:21065 > > CLOSE_WAIT - keepalive (7164.63/0/0) > > ... > > > > > > On Tomcat side: > > > > (Not all processes could be identified, non-owned process info > > will not be shown, you would have to be root to see it all.) > > Active Internet connections (servers and established) > > Proto Recv-Q Send-Q Local Address Foreign Address > > State PID/Program nameTimer > > ... > > tcp0 0 :::21065:::* > > LISTEN 6988/java off (0.00/0/0) > > tcp0 0 :::127.0.0.1:11065 :::* > > LISTEN 6988/java off (0.00/0/0) > > tcp0 0 :::172.27.127.201:21065 :::129.41.29.243:43992 > > FIN_WAIT2 - timewait (56.71/0/0) > > tcp0 0 :::172.27.127.201:21065 :::129.41.29.243:43991 > > FIN_WAIT2 - timewait (56.24/0/0) > > ... > > > > > > > [Pantvaidya, Vishwajit] By the way, in the thread console, I see 8 > TP-Processor threads (2 RUNNABLE, 6 WAITING). But above netstat output on > tomcat side shows only 2 connections
RE: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity
> > The fact that *none* of the ports match would suggest (but not prove) > that > > someone in the middle is closing the connections, and not telling either > > end about it. > > > > Do the netstat -anop again; it should be more interesting. > > > > - Chuck > > > > [Pantvaidya, Vishwajit] Tomcat server port 11065, connector port 21065 > > On Httpd Side: > > Proto Recv-Q Send-Q Local Address Foreign Address > State PID/Program nameTimer > ... > tcp0 0 0.0.0.0:25 0.0.0.0:* > LISTEN - off (0.00/0/0) > tcp1 0 129.41.29.243:44003 172.27.127.201:21065 > CLOSE_WAIT - keepalive (7194.80/0/0) > tcp1 0 129.41.29.243:44002 172.27.127.201:21065 > CLOSE_WAIT - keepalive (7194.43/0/0) > tcp1 0 129.41.29.243:44001 172.27.127.201:21065 > CLOSE_WAIT - keepalive (7192.26/0/0) > tcp1 0 129.41.29.243:44000 172.27.127.201:21065 > CLOSE_WAIT - keepalive (7189.64/0/0) > tcp1 0 129.41.29.243:43990 172.27.127.201:21065 > CLOSE_WAIT - keepalive (7016.23/0/0) > tcp1 0 129.41.29.243:43999 172.27.127.201:21065 > CLOSE_WAIT - keepalive (7189.30/0/0) > tcp1 0 129.41.29.243:43998 172.27.127.201:21065 > CLOSE_WAIT - keepalive (7186.76/0/0) > tcp1 0 129.41.29.243:43996 172.27.127.201:21065 > CLOSE_WAIT - keepalive (7183.86/0/0) > tcp1 0 129.41.29.243:43994 172.27.127.201:21065 > CLOSE_WAIT - keepalive (7174.09/0/0) > tcp1 0 129.41.29.243:43993 172.27.127.201:21065 > CLOSE_WAIT - keepalive (7164.63/0/0) > ... > > > On Tomcat side: > > (Not all processes could be identified, non-owned process info > will not be shown, you would have to be root to see it all.) > Active Internet connections (servers and established) > Proto Recv-Q Send-Q Local Address Foreign Address > State PID/Program nameTimer > ... > tcp0 0 :::21065:::* > LISTEN 6988/java off (0.00/0/0) > tcp0 0 :::127.0.0.1:11065 :::* > LISTEN 6988/java off (0.00/0/0) > tcp0 0 :::172.27.127.201:21065 :::129.41.29.243:43992 > FIN_WAIT2 - timewait (56.71/0/0) > tcp0 0 :::172.27.127.201:21065 :::129.41.29.243:43991 > FIN_WAIT2 - timewait (56.24/0/0) > ... > > > [Pantvaidya, Vishwajit] By the way, in the thread console, I see 8 TP-Processor threads (2 RUNNABLE, 6 WAITING). But above netstat output on tomcat side shows only 2 connections on port 21065. Shouldn't there be 1 thread / connection? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity
> The fact that *none* of the ports match would suggest (but not prove) that > someone in the middle is closing the connections, and not telling either > end about it. > > Do the netstat -anop again; it should be more interesting. > > - Chuck > [Pantvaidya, Vishwajit] Tomcat server port 11065, connector port 21065 On Httpd Side: Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program nameTimer ... tcp0 0 0.0.0.0:25 0.0.0.0:* LISTEN - off (0.00/0/0) tcp1 0 129.41.29.243:44003 172.27.127.201:21065 CLOSE_WAIT - keepalive (7194.80/0/0) tcp1 0 129.41.29.243:44002 172.27.127.201:21065 CLOSE_WAIT - keepalive (7194.43/0/0) tcp1 0 129.41.29.243:44001 172.27.127.201:21065 CLOSE_WAIT - keepalive (7192.26/0/0) tcp1 0 129.41.29.243:44000 172.27.127.201:21065 CLOSE_WAIT - keepalive (7189.64/0/0) tcp1 0 129.41.29.243:43990 172.27.127.201:21065 CLOSE_WAIT - keepalive (7016.23/0/0) tcp1 0 129.41.29.243:43999 172.27.127.201:21065 CLOSE_WAIT - keepalive (7189.30/0/0) tcp1 0 129.41.29.243:43998 172.27.127.201:21065 CLOSE_WAIT - keepalive (7186.76/0/0) tcp1 0 129.41.29.243:43996 172.27.127.201:21065 CLOSE_WAIT - keepalive (7183.86/0/0) tcp1 0 129.41.29.243:43994 172.27.127.201:21065 CLOSE_WAIT - keepalive (7174.09/0/0) tcp1 0 129.41.29.243:43993 172.27.127.201:21065 CLOSE_WAIT - keepalive (7164.63/0/0) ... On Tomcat side: (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program nameTimer ... tcp0 0 :::21065:::* LISTEN 6988/java off (0.00/0/0) tcp0 0 :::127.0.0.1:11065 :::* LISTEN 6988/java off (0.00/0/0) tcp0 0 :::172.27.127.201:21065 :::129.41.29.243:43992 FIN_WAIT2 - timewait (56.71/0/0) tcp0 0 :::172.27.127.201:21065 :::129.41.29.243:43991 FIN_WAIT2 - timewait (56.24/0/0) ... - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity
> From: Pantvaidya, Vishwajit [mailto:vpant...@selectica.com] > Subject: RE: Running out of tomcat threads - why many threads in > RUNNABLEstage even with no activity > > On httpd machine > Proto Recv-Q Send-Q Local Address Foreign Address > State > tcp1 0 129.41.29.243:43225 172.27.127.201:21065 > CLOSE_WAIT > tcp1 0 129.41.29.243:43227 172.27.127.201:21065 > CLOSE_WAIT > tcp1 0 129.41.29.243:43237 172.27.127.201:21065 > CLOSE_WAIT > tcp1 0 129.41.29.243:43244 172.27.127.201:21065 > CLOSE_WAIT > tcp1 0 129.41.29.243:43245 172.27.127.201:21065 > CLOSE_WAIT > > On tomcat machine > Proto Recv-Q Send-Q Local Address Foreign Address > State > tcp0 0 :::172.27.127.201:21065 :::129.41.29.243:43204 > TIME_WAIT > tcp0 0 :::172.27.127.201:21065 :::129.41.29.243:43205 > TIME_WAIT > tcp0 0 :::172.27.127.201:21065 :::129.41.29.243:43206 > TIME_WAIT > tcp0 0 :::172.27.127.201:21065 :::129.41.29.243:43211 > TIME_WAIT > tcp0 0 :::172.27.127.201:21065 :::129.41.29.243:43212 > FIN_WAIT2 > tcp0 0 :::172.27.127.201:21065 :::129.41.29.243:43213 > FIN_WAIT2 > tcp0 0 :::172.27.127.201:21065 :::129.41.29.243:43214 > FIN_WAIT2 > tcp0 0 :::172.27.127.201:21065 :::129.41.29.243:43215 > TIME_WAIT > tcp0 0 :::172.27.127.201:21065 :::129.41.29.243:43216 > FIN_WAIT2 > tcp0 0 :::172.27.127.201:21065 :::129.41.29.243:43217 > FIN_WAIT2 > tcp0 0 :::172.27.127.201:21065 :::129.41.29.243:43218 > FIN_WAIT2 (The above was edited to remove irrelevant IP addresses and sort by port.) The fact that *none* of the ports match would suggest (but not prove) that someone in the middle is closing the connections, and not telling either end about it. > - why do the 11 threads in the httpd o/p show port > 21069 in foreign addr. They're for a different IP address. > - currently I do have connectionTimeout set in server.xml. > I will need to wait until night to reset that. Do the netstat -anop again; it should be more interesting. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity
> From: André Warnier [mailto:a...@ice-sa.com] > Subject: Re: Running out of tomcat threads - why many threads in > RUNNABLEstage even with no activity > if you add "-p" to netstat (at least under Linux), it will also > show the program that corresponds to that line. Or at least -o to show the pid number. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity
Pantvaidya, Vishwajit wrote: - why do the 11 threads in the httpd o/p show port 21069 in foreign addr. Or are those not the correct threads I should be looking at?a A bit off-topic maybe, but if you add "-p" to netstat (at least under Linux), it will also show the program that corresponds to that line. That may help checking if you are looking at the correct ones. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity
> -Original Message- > From: Rainer Jung [mailto:rainer.j...@kippdata.de] > Sent: Wednesday, May 20, 2009 11:53 AM > To: Tomcat Users List > Subject: Re: Running out of tomcat threads - why many threads in RUNNABLE > stage even with no activity > > On 20.05.2009 19:47, Caldarale, Charles R wrote: > >> From: Caldarale, Charles R [mailto:chuck.caldar...@unisys.com] > >> Subject: RE: Running out of tomcat threads - why many threads > >> inRUNNABLEstage even with no activity > >> > >>>> - Setting connectionTimeout in server.xml seems to have resolved > >>>> the issue > >> Only because you're throwing away what appears to be a usable > >> connection that's designed to be persistent. > > > > Do you have something between Tomcat and httpd that could be silently > closing connections? (Some badly behaved firewalls are known to do this.) > That would make the existing AJP connections useless, without notifying > the Tomcat threads that the connection is no longer there. Setting the > timeout would allow those connections to be discarded and new ones > created. > > That's a good point. You should check both sides by using "netstat -an". > The Apache side and the Tomcat side (without connectionTimeout, so you > can see the problem in the original form). See whether the number of AJP > connections in the various TCP states differs much between the netstat > output on the Apache and on the Tomcat system. > [Pantvaidya, Vishwajit] My tomcat connector port is 21065. Netstat shows following output under "Active Internet Connections": On httpd machine Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State tcp0 0 0.0.0.0:25 0.0.0.0:* LISTEN tcp1 0 129.41.29.243:43237 172.27.127.201:21065 CLOSE_WAIT tcp1 0 129.41.29.243:43244 172.27.127.201:21065 CLOSE_WAIT tcp1 0 129.41.29.243:43245 172.27.127.201:21065 CLOSE_WAIT tcp1 0 129.41.29.243:43225 172.27.127.201:21065 CLOSE_WAIT tcp1 0 129.41.29.243:43227 172.27.127.201:21065 CLOSE_WAIT tcp0 0 129.41.29.243:43239 172.27.127.202:21069 ESTABLISHED tcp0 0 129.41.29.243:43238 172.27.127.202:21069 ESTABLISHED tcp0 0 129.41.29.243:43243 172.27.127.202:21069 ESTABLISHED tcp0 0 129.41.29.243:43242 172.27.127.202:21069 ESTABLISHED tcp0 0 129.41.29.243:43241 172.27.127.202:21069 ESTABLISHED tcp0 0 129.41.29.243:43240 172.27.127.202:21069 ESTABLISHED tcp0 0 129.41.29.243:43247 172.27.127.202:21069 ESTABLISHED tcp0 0 129.41.29.243:43246 172.27.127.202:21069 ESTABLISHED tcp0 0 129.41.29.243:43219 172.27.127.202:21069 TIME_WAIT tcp0 0 129.41.29.243:43209 172.27.127.202:21069 ESTABLISHED tcp0 0 129.41.29.243:43208 172.27.127.202:21069 TIME_WAIT On tomcat machine Proto Recv-Q Send-Q Local Address Foreign Address State tcp0 0 :::172.27.127.201:21065 :::129.41.29.243:43216 FIN_WAIT2 tcp0 0 :::172.27.127.201:21065 :::129.41.29.243:43217 FIN_WAIT2 tcp0 0 :::172.27.127.201:21065 :::129.41.29.243:43218 FIN_WAIT2 tcp0 0 :::172.27.127.201:21065 :::129.41.29.243:43211 TIME_WAIT tcp0 0 :::172.27.127.201:21065 :::129.41.29.243:43212 FIN_WAIT2 tcp0 0 :::172.27.127.201:21065 :::129.41.29.243:43213 FIN_WAIT2 tcp0 0 :::172.27.127.201:21065 :::129.41.29.243:43214 FIN_WAIT2 tcp0 0 :::172.27.127.201:21065 :::129.41.29.243:43215 TIME_WAIT tcp0 0 :::172.27.127.201:21065 :::129.41.29.243:43204 TIME_WAIT tcp0 0 :::172.27.127.201:21065 :::129.41.29.243:43205 TIME_WAIT tcp0 0 :::172.27.127.201:21065 :::129.41.29.243:43206 TIME_WAIT - My thread dump shows 8 TP-Processor threads - but this output has 11 threads. - why do the 11 threads in the httpd o/p show port 21069 in foreign addr. Or are those not the correct threads I should be looking at? - currently I do have connectionTimeout set in server.xml. I will need to wait until night to reset that. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity
> > > > Finally, is it possible that some bad code in the app could > > be hanging onto those RUNNABLE connections which is why tomcat > > is not releasing them? > > Once more: NO, NO, NO! The threads you see in a RUNNABLE state are > perfectly normal and expected. Go do the netstat that Rainer suggested > and let us know what you see. Stop fixating on the thread state. > > - Chuck > > [Pantvaidya, Vishwajit] Ok will do Chuck - thanks a lot for persisting with me through this issue. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity
> From: Pantvaidya, Vishwajit [mailto:vpant...@selectica.com] > Subject: RE: Running out of tomcat threads - why many threads in > RUNNABLEstage even with no activity > > Finally, is it possible that some bad code in the app could > be hanging onto those RUNNABLE connections which is why tomcat > is not releasing them? Once more: NO, NO, NO! The threads you see in a RUNNABLE state are perfectly normal and expected. Go do the netstat that Rainer suggested and let us know what you see. Stop fixating on the thread state. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity
> -Original Message- > From: Rainer Jung [mailto:rainer.j...@kippdata.de] > Sent: Wednesday, May 20, 2009 11:53 AM > To: Tomcat Users List > Subject: Re: Running out of tomcat threads - why many threads in RUNNABLE > stage even with no activity > > On 20.05.2009 19:47, Caldarale, Charles R wrote: > >> From: Caldarale, Charles R [mailto:chuck.caldar...@unisys.com] > >> Subject: RE: Running out of tomcat threads - why many threads > >> inRUNNABLEstage even with no activity > >> > >>>> - Setting connectionTimeout in server.xml seems to have resolved > >>>> the issue > >> Only because you're throwing away what appears to be a usable > >> connection that's designed to be persistent. > > > > Do you have something between Tomcat and httpd that could be silently > closing connections? (Some badly behaved firewalls are known to do this.) > That would make the existing AJP connections useless, without notifying > the Tomcat threads that the connection is no longer there. Setting the > timeout would allow those connections to be discarded and new ones > created. > > That's a good point. You should check both sides by using "netstat -an". > The Apache side and the Tomcat side (without connectionTimeout, so you > can see the problem in the original form). See whether the number of AJP > connections in the various TCP states differs much between the netstat > output on the Apache and on the Tomcat system. > [Pantvaidya, Vishwajit] Ok will do this. To complicate things - we are seeing these outofthread problems only in one of our production servers - so I need to figure out if there are any differences in firewall settings between the 2 servers. Finally, is it possible that some bad code in the app could be hanging onto those RUNNABLE connections which is why tomcat is not releasing them? Or if that was the case, would the stack trace of that thread show the code that was hanging onto it? In my case, all RUNNABLE connections show the stacktrace "TP-Processor4" - Thread t...@29 java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method)... at org.apache.jk.common.ChannelSocket.accept(ChannelSocket.java:293)... at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684) at java.lang.Thread.run(Thread.java:595) - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity
> > > > [Pantvaidya, Vishwajit] My problem is that tomcat is running out of > threads (maxthreadcount=200). My analysis of the issue is: > > - threads count is exceeded because of a slow buildup of RUNNABLE > threads (and not because number of simultaneous http requests at some > point exceeded max thread count) > > I don't belibve this reason. I would say thread count is exceeded, > because you allow a much higher concurrency on the web server layer. > [Pantvaidya, Vishwajit] Is there a tool you can recommend for me to monitor/log the http requests so that I have figures to back up my analysis. > > - most/all newly created TP-Processor threads are in RUNNABLE state and > remain RUNNABLE - never go back to WAITING state (waiting for thread pool) > > So you are using persistent connections. There's no *problem* with that > per se. If you ae uncomfortable with it configure the timeouts in the > Tomcat connector *and* mod_jk. > [Pantvaidya, Vishwajit] Ok so RUNNABLE i.e. persistent threads should not be an issue. The only reason why I thought that was an issue was that I was observing that the none of the RUNNABLE connections were not being used to serve new requests, only the WAITING ones were - and I do know for sure that the RUNNABLE threads were not servicing any existing requests as I was the only one using the system then. > > - in such case, I find that tomcat spawns new threads when a new request > comes in > > "request" -> "connection" > > > - this continues and finally tomcat runs out of threads > > That's to simple, usually the "new requests" should be handled by > existing Apache processes that already have a connection to Tomcat and > will not create a new one. > [Pantvaidya, Vishwajit] In my case the existing persistent connections are not servicing any new requests. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity
On 20.05.2009 19:47, Caldarale, Charles R wrote: >> From: Caldarale, Charles R [mailto:chuck.caldar...@unisys.com] >> Subject: RE: Running out of tomcat threads - why many threads >> inRUNNABLEstage even with no activity >> >>>> - Setting connectionTimeout in server.xml seems to have resolved >>>> the issue >> Only because you're throwing away what appears to be a usable >> connection that's designed to be persistent. > > Do you have something between Tomcat and httpd that could be silently closing > connections? (Some badly behaved firewalls are known to do this.) That > would make the existing AJP connections useless, without notifying the Tomcat > threads that the connection is no longer there. Setting the timeout would > allow those connections to be discarded and new ones created. That's a good point. You should check both sides by using "netstat -an". The Apache side and the Tomcat side (without connectionTimeout, so you can see the problem in the original form). See whether the number of AJP connections in the various TCP states differs much between the netstat output on the Apache and on the Tomcat system. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity
> From: Caldarale, Charles R [mailto:chuck.caldar...@unisys.com] > Subject: RE: Running out of tomcat threads - why many threads > inRUNNABLEstage even with no activity > > > > - Setting connectionTimeout in server.xml seems to have resolved > > > the issue > > Only because you're throwing away what appears to be a usable > connection that's designed to be persistent. Do you have something between Tomcat and httpd that could be silently closing connections? (Some badly behaved firewalls are known to do this.) That would make the existing AJP connections useless, without notifying the Tomcat threads that the connection is no longer there. Setting the timeout would allow those connections to be discarded and new ones created. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Running out of tomcat threads - why many threads inRUNNABLEstage even with no activity
> From: Pantvaidya, Vishwajit [mailto:vpant...@selectica.com] > Subject: RE: Running out of tomcat threads - why many threads > inRUNNABLEstage even with no activity > > > [Pantvaidya, Vishwajit] Thanks Rainer. The RUNNABLE thread - > > is it a connection between Tomcat and webserver, or between > > Tomcat and AJP? It's not at all clear what you mean by that. AJP is a protocol used by Tomcat and a front-end webserver, such as httpd. > > Is it still RUNNABLE and not WAITING because the servlet has > > not explicitly closed the connection yet (something like > > HttpServletResponse.getOutputStresm.close) No, the RUNNABLE is the normal state when the thread is not processing a request. The connections between Tomcat and httpd are persistent - they're never meant to close. > > [Pantvaidya, Vishwajit] My problem is that tomcat is running out of > > threads (maxthreadcount=200). Perhaps you provided some evidence earlier, but I missed it. Having threads in a RUNNABLE state waiting for requests to arrive from httpd is *normal* and expected; it's *not* a problem. You need to stop treating it like it's some terrible condition - it's how things should be. > > - Setting connectionTimeout in server.xml seems to have resolved > > the issue Only because you're throwing away what appears to be a usable connection that's designed to be persistent. > 1. Is it expected behavior that most tomcat threads are in > RUNNABLE state? As we keep telling you, yes. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity
On 20.05.2009 00:53, Pantvaidya, Vishwajit wrote: >> -Original Message- >> From: Rainer Jung [mailto:rainer.j...@kippdata.de] >> Sent: Monday, May 18, 2009 11:10 PM >> To: Tomcat Users List >> Subject: Re: Running out of tomcat threads - why many threads in >> RUNNABLEstage even with no activity >> >> On 19.05.2009 02:54, Caldarale, Charles R wrote: >>>> From: Pantvaidya, Vishwajit [mailto:vpant...@selectica.com] >>>> Subject: RE: Running out of tomcat threads - why many threads in >>>> RUNNABLEstage even with no activity >>>> >>>> Ok - so then the question is when does tomcat transition the thread >>>> from Running to Waiting? Does that happen after AJP drops that >>>> connection? >> RUNNABLE and WAITING are thread states in the JVM. They don't relate in >> general to states inside Tomcat. In this special situation they do. >> >> The states you observe are both completely normal in themselves. One >> (the stack you abbreviate with RUNNABLE) is handling a persistant >> connection between web server and Tomcat which could send more requests, >> but at the moment no request is being processed, the other (you >> abbreviate with WAITING) is available to be associated with a new >> connection that might come in some time in the future. >> > > [Pantvaidya, Vishwajit] Thanks Rainer. The RUNNABLE thread - is it a > connection between Tomcat and webserver, or between Tomcat and AJP? Is it > still RUNNABLE and not WAITING because the servlet has not explicitly closed > the connection yet (something like HttpServletResponse.getOutputStresm.close) The thread handles a connection betwen the web server and Tomcat. AJP is the protocol used on that connection. It is runnable, because a socket read from inside the JVM puts a thread into runnable state. The socket read is used to read the next request and will block until data arrives over the established connection. >>>> So could the problem be occurring here because AJP is holding on to >>>> connections? >>> Sorry, I haven't been following the thread that closely. Not sure >>> what the problem you're referring to actually is, but having a Tomcat >>> thread reading input from the AJP connector is pretty normal. >> The same to me. What's the problem? AJP is designed to reuse connections >> (use persistent connections). If you do not want them to be used for a >> very long time or like those connections to be closed when being idle, >> you have to configure the appropriate timeouts. Look at the timeouts >> documentation page of mod_jk. >> >> In general your max thread numbers in the web server layer and in the >> Tomcat AJP pool need to be set consistently. >> > > [Pantvaidya, Vishwajit] My problem is that tomcat is running out of threads > (maxthreadcount=200). My analysis of the issue is: > - threads count is exceeded because of a slow buildup of RUNNABLE threads > (and not because number of simultaneous http requests at some point exceeded > max thread count) I don't belibve this reason. I would say thread count is exceeded, because you allow a much higher concurrency on the web server layer. > - most/all newly created TP-Processor threads are in RUNNABLE state and > remain RUNNABLE - never go back to WAITING state (waiting for thread pool) So you are using persistent connections. There's no *problem* with that per se. If you ae uncomfortable with it configure the timeouts in the Tomcat connector *and* mod_jk. > - in such case, I find that tomcat spawns new threads when a new request > comes in "request" -> "connection" > - this continues and finally tomcat runs out of threads That's to simple, usually the "new requests" should be handled by existing Apache processes that already have a connection to Tomcat and will not create a new one. > - Setting connectionTimeout in server.xml seems to have resolved the issue - > but I am wondering if that was just a workaround i.e. whether so many threads > remaining RUNNABLE indicate a flaw in our webapp i.e. it not doing whatever's > necessary to close them and return them to WAITING state. No it is a misconifguration of your web server, mod_jk and Tomcat. The use of persistent AJP connections is opaque to the web application. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity
> > RUNNABLE and WAITING are thread states in the JVM. They don't relate in > > general to states inside Tomcat. In this special situation they do. > > > > The states you observe are both completely normal in themselves. One > > (the stack you abbreviate with RUNNABLE) is handling a persistant > > connection between web server and Tomcat which could send more requests, > > but at the moment no request is being processed, the other (you > > abbreviate with WAITING) is available to be associated with a new > > connection that might come in some time in the future. > > > > [Pantvaidya, Vishwajit] Thanks Rainer. The RUNNABLE thread - is it a > connection between Tomcat and webserver, or between Tomcat and AJP? Is it > still RUNNABLE and not WAITING because the servlet has not explicitly > closed the connection yet (something like > HttpServletResponse.getOutputStresm.close) > > > [Pantvaidya, Vishwajit] My problem is that tomcat is running out of > threads (maxthreadcount=200). My analysis of the issue is: > - threads count is exceeded because of a slow buildup of RUNNABLE threads > (and not because number of simultaneous http requests at some point > exceeded max thread count) > - most/all newly created TP-Processor threads are in RUNNABLE state and > remain RUNNABLE - never go back to WAITING state (waiting for thread pool) > - in such case, I find that tomcat spawns new threads when a new request > comes in > - this continues and finally tomcat runs out of threads > - Setting connectionTimeout in server.xml seems to have resolved the issue > - but I am wondering if that was just a workaround i.e. whether so many > threads remaining RUNNABLE indicate a flaw in our webapp i.e. it not doing > whatever's necessary to close them and return them to WAITING state. > [Pantvaidya, Vishwajit] After setting connectionTimeout in tomcat server.xml, the number of open threads is now consistently under 10 and most of them are now in WAITING stage. So looks like connectionTimeout also destroys idle threads. But I am still wondering - why should I have to set connectionTimeout to prevent tomcat running out of threads. I certainly don't mind if the TP-Processor threads continue to hang around as long as they are in WAITING state. 1. Is it expected behavior that most tomcat threads are in RUNNABLE state? 2. If not, does it indicate a problem in the app or in tomcat configuration? My thinking is that the answer to #1 is no, and that to #2 is that it is an app problem. But just wanted to confirm and find out what people out there think. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity
> -Original Message- > From: Rainer Jung [mailto:rainer.j...@kippdata.de] > Sent: Monday, May 18, 2009 11:10 PM > To: Tomcat Users List > Subject: Re: Running out of tomcat threads - why many threads in > RUNNABLEstage even with no activity > > On 19.05.2009 02:54, Caldarale, Charles R wrote: > >> From: Pantvaidya, Vishwajit [mailto:vpant...@selectica.com] > >> Subject: RE: Running out of tomcat threads - why many threads in > >> RUNNABLEstage even with no activity > >> > >> Ok - so then the question is when does tomcat transition the thread > >> from Running to Waiting? Does that happen after AJP drops that > >> connection? > > RUNNABLE and WAITING are thread states in the JVM. They don't relate in > general to states inside Tomcat. In this special situation they do. > > The states you observe are both completely normal in themselves. One > (the stack you abbreviate with RUNNABLE) is handling a persistant > connection between web server and Tomcat which could send more requests, > but at the moment no request is being processed, the other (you > abbreviate with WAITING) is available to be associated with a new > connection that might come in some time in the future. > [Pantvaidya, Vishwajit] Thanks Rainer. The RUNNABLE thread - is it a connection between Tomcat and webserver, or between Tomcat and AJP? Is it still RUNNABLE and not WAITING because the servlet has not explicitly closed the connection yet (something like HttpServletResponse.getOutputStresm.close) > > > >> So could the problem be occurring here because AJP is holding on to > >> connections? > > > Sorry, I haven't been following the thread that closely. Not sure > > what the problem you're referring to actually is, but having a Tomcat > > thread reading input from the AJP connector is pretty normal. > > The same to me. What's the problem? AJP is designed to reuse connections > (use persistent connections). If you do not want them to be used for a > very long time or like those connections to be closed when being idle, > you have to configure the appropriate timeouts. Look at the timeouts > documentation page of mod_jk. > > In general your max thread numbers in the web server layer and in the > Tomcat AJP pool need to be set consistently. > [Pantvaidya, Vishwajit] My problem is that tomcat is running out of threads (maxthreadcount=200). My analysis of the issue is: - threads count is exceeded because of a slow buildup of RUNNABLE threads (and not because number of simultaneous http requests at some point exceeded max thread count) - most/all newly created TP-Processor threads are in RUNNABLE state and remain RUNNABLE - never go back to WAITING state (waiting for thread pool) - in such case, I find that tomcat spawns new threads when a new request comes in - this continues and finally tomcat runs out of threads - Setting connectionTimeout in server.xml seems to have resolved the issue - but I am wondering if that was just a workaround i.e. whether so many threads remaining RUNNABLE indicate a flaw in our webapp i.e. it not doing whatever's necessary to close them and return them to WAITING state. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity
> > Yes, sure - we will upgrade at some point of > > time. But since upgrading all our servers will be some work, that may > > not happen right away. > > Upgrading mod_jk is the least painful of all of these, and the most > likely to affect you. > [Pantvaidya, Vishwajit] I understand and agree and will push for this - but most admins are conservative and so I am not harboring high hopes for success in this. > > Here are figures from 3 of the servers which > > have not yet run out of threads (so the thread count does not add up > > to 200). I have taken these late at night when no users are present, > > so I was expecting all threads to be Waiting for tomcat thread-pool. > > > > 1. Total TP-Processor threads 48, Waiting 46, Runnable 2 > > 2. Total TP-Processor threads 40, Waiting 29, Runnable 11 > > 3. Total TP-Processor threads 120, Waiting 7, Runnable 113 > > Are you sure you aren't seeing any traffic, even that late at night? > What if you watch the access logs? Are there requests actively being > serviced? > [Pantvaidya, Vishwajit] I was "tail"ing the logs - there were no accesses. > > Do you think this could be because of the > > application? I was under the impression that there is some tomcat > > config parameter that controls this - which was set to 4. > > No, Tomcat uses precisely 1 thread to handle each incoming HTTP request. > If keepalives are used, multiple requests may be handled by the same > thread before it is returned to the pool, or different threads may be > used to serve different requests from the single connection, but in > either case, no more than 1 thread will be used to service a single HTTP > request. > [Pantvaidya, Vishwajit] Could this happen if upon my http browser request, the app could be spawning multiple redirects in quick succession leading tomcat to create multiple threads. Any other thoughts why I could be seeing tomcat spawn threads in multiples of 4? > >>> My workers config is: > >>> > >>> Worker...type=ajp13 > >>> Worker...cachesize=10 > >> Are you using the prefork MPM? If so, cachesize should be /1/. > > > > [Pantvaidya, Vishwajit] Could you please elaborate. What is the > > prefork MPM? > > The MPM is the concurrency strategy employed by Apache httpd. Either you > are using the "prefork MPM" which starts multiple httpd processes to > handle requests, or you are using the "worker MPM" which starts multiple > threads to handle requests. Actually, mod_jk should be able to > auto-detect the appropriate cachesize (called connection_pool_size, > now), so you shouldn't have to set this. > [Pantvaidya, Vishwajit] Ok thanks. "httpd -l" showed perfork.c. I guess that means we are using prefork MPM. So our cachesize should be 1? Our mod_jk version is 1.2.15 - will that also auto-detect the cache-size? > >>> Worker...cache_timeout=600 Worker...socket_keepalive=1 > >>> Worker...recycle_timeout=300 > >> Are these timeouts necessary? Why not simply let the connections > >> stay alive all the time? > >> > > [Pantvaidya, Vishwajit] Sure we could. But for any production change, > > I would have to offer a good enough reason. > > What was the "good enough reason" to set those timeouts in the first > place? > [Pantvaidya, Vishwajit] I agree - but again as I mentioned above because the admin will be conservative about any changes, I need to have a strong reason. Also when you say let the connection stay alive, does that mean let the TP-Processor thread remain in Waiting state / Runnable state? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Running out of tomcat threads - why many threads in RUNNABLEstageeven with no activity
> From: Christopher Schultz [mailto:ch...@christopherschultz.net] > Subject: Re: Running out of tomcat threads - why many threads in > RUNNABLEstageeven with no activity > > Or, maybe Java's thread states don't differentiate between > actually runnable and runnable but blocked (as opposed to > WAITING which means waiting on a synchronization monitor). Correct; there is no BLOCKED state from the point of view of a Java thread. When in native code, such as reading from a socket or any other JNI method, the thread is still considered RUNNABLE, since the VM has no control over when it might be executing. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
Re: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Chuck, On 5/18/2009 8:54 PM, Caldarale, Charles R wrote: >> From: Pantvaidya, Vishwajit [mailto:vpant...@selectica.com] >> Subject: RE: Running out of tomcat threads - why many threads in >> RUNNABLEstage even with no activity >> >> Ok - so then the question is when does tomcat transition the thread >> from Running to Waiting? Does that happen after AJP drops that >> connection? > > That's my understanding; I would presume some from of keep-alive is > in play. However, others know the AJP characteristics better than I > do. Rainer is the ultimate resource, but I suspect he's asleep right > now. My expectation would be that an AJP connection waiting for the "next" request in a set of keepalive requests would be WAITING: blocked on a socket read, rather than RUNNABLE. Or, maybe Java's thread states don't differentiate between actually runnable and runnable but blocked (as opposed to WAITING which means waiting on a synchronization monitor). - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkoSpdcACgkQ9CaO5/Lv0PAcnwCgq33fogBqYaYD5INtQk8D/x7d RewAn23Ft0nSsgSQeupKhuanWdlwsIsS =jHcT -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity
On 19.05.2009 02:54, Caldarale, Charles R wrote: >> From: Pantvaidya, Vishwajit [mailto:vpant...@selectica.com] >> Subject: RE: Running out of tomcat threads - why many threads in >> RUNNABLEstage even with no activity >> >> Ok - so then the question is when does tomcat transition the thread >> from Running to Waiting? Does that happen after AJP drops that >> connection? RUNNABLE and WAITING are thread states in the JVM. They don't relate in general to states inside Tomcat. In this special situation they do. The states you observe are both completely normal in themselves. One (the stack you abbreviate with RUNNABLE) is handling a persistant connection between web server and Tomcat which could send more requests, but at the moment no request is being processed, the other (you abbreviate with WAITING) is available to be associated with a new connection that might come in some time in the future. > That's my understanding; I would presume some from of keep-alive is > in play. However, others know the AJP characteristics better than I HTTP Keep-Alive does not change the picture. It's transparent to Tomcat and mod_jk. Those Keep-Alice packets do not count as requests. > do. Rainer is the ultimate resource, but I suspect he's asleep right > now. > >> So could the problem be occurring here because AJP is holding on to >> connections? > Sorry, I haven't been following the thread that closely. Not sure > what the problem you're referring to actually is, but having a Tomcat > thread reading input from the AJP connector is pretty normal. The same to me. What's the problem? AJP is designed to reuse connections (use persistent connections). If you do not want them to be used for a very long time or like those connections to be closed when being idle, you have to configure the appropriate timeouts. Look at the timeouts documentation page of mod_jk. In general your max thread numbers in the web server layer and in the Tomcat AJP pool need to be set consistently. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity
> From: Pantvaidya, Vishwajit [mailto:vpant...@selectica.com] > Subject: RE: Running out of tomcat threads - why many threads in > RUNNABLEstage even with no activity > > Ok - so then the question is when does tomcat transition > the thread from Running to Waiting? Does that happen after > AJP drops that connection? That's my understanding; I would presume some from of keep-alive is in play. However, others know the AJP characteristics better than I do. Rainer is the ultimate resource, but I suspect he's asleep right now. > So could the problem be occurring here because AJP is > holding on to connections? Sorry, I haven't been following the thread that closely. Not sure what the problem you're referring to actually is, but having a Tomcat thread reading input from the AJP connector is pretty normal. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity
> -Original Message- > From: Caldarale, Charles R [mailto:chuck.caldar...@unisys.com] > Sent: Monday, May 18, 2009 4:02 PM > To: Tomcat Users List > Subject: RE: Running out of tomcat threads - why many threads in > RUNNABLEstage even with no activity > > > From: Pantvaidya, Vishwajit [mailto:vpant...@selectica.com] > > Subject: RE: Running out of tomcat threads - why many threads in > > RUNNABLEstage even with no activity > > > > From whatever I have read on this, it seems to me that this could > > happen if a servlet writes something to a response stream, closes > > the response stream, but after that keeps on doing some processing > > (e.g. running an infinite loop). > > No - the thread would be inside the servlet in that case. The thread here > in the RUNNABLE state is waiting for a *new* request to come in over an > active AJP connection; a thread in the WAITING state would be assigned to > a new connection when one is accepted. > [Pantvaidya, Vishwajit] Ok - so then the question is when does tomcat transition the thread from Running to Waiting? Does that happen after AJP drops that connection? So could the problem be occurring here because AJP is holding on to connections? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity
> From: Pantvaidya, Vishwajit [mailto:vpant...@selectica.com] > Subject: RE: Running out of tomcat threads - why many threads in > RUNNABLEstage even with no activity > > From whatever I have read on this, it seems to me that this could > happen if a servlet writes something to a response stream, closes > the response stream, but after that keeps on doing some processing > (e.g. running an infinite loop). No - the thread would be inside the servlet in that case. The thread here in the RUNNABLE state is waiting for a *new* request to come in over an active AJP connection; a thread in the WAITING state would be assigned to a new connection when one is accepted. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity
> -Original Message- > From: Rainer Jung [mailto:rainer.j...@kippdata.de] > Sent: Monday, May 18, 2009 2:43 PM > To: Tomcat Users List > Subject: Re: Running out of tomcat threads - why many threads in RUNNABLE > stage even with no activity > > Yes, those two look like "waiting for next request on an existing > connection from the web server to Tomcat" and "sitting idle in the pool, > waiting for a new connection to handle". > [Pantvaidya, Vishwajit] Thanks Rainier. Any idea why threads would be sitting around in Runnable state even when nobody has been using application for a long time. From whatever I have read on this, it seems to me that this could happen if a servlet writes something to a response stream, closes the response stream, but after that keeps on doing some processing (e.g. running an infinite loop). I am reasonably sure that our app is not doing something like that. Unless there was something like an infinite loop running in a servlet, I would assume that the servlet would eventually return and the tomcat TP-Processor thread would be released back to the connection pool (go into Waiting state). > On 18.05.2009 22:44, Pantvaidya, Vishwajit wrote: > >>> [Pantvaidya, Vishwajit] Here are figures from 3 of the servers which > > > >> have > > > >>> not yet run out of threads (so the thread count does not add up to > 200). > > > >> I > > > >>> have taken these late at night when no users are present, so I was > > > >>> expecting all threads to be Waiting for tomcat thread-pool. > > > > > >>> 1. Total TP-Processor threads 48, Waiting 46, Runnable 2 > > > >>> 2. Total TP-Processor threads 40, Waiting 29, Runnable 11 > > > >>> 3. Total TP-Processor threads 120, Waiting 7, Runnable 113 > > > > > > > > [Pantvaidya, Vishwajit] Since Rainer mentioned that he would like to see > more of the stack trace, here are the complete stack traces for a Runnable > and Waiting thread from #3 above. All Runnable/Waiting threads from all > the above cases have same stack trace as below: > > > > > > > > "TP-Processor119" - Thread t...@2294 > > > >java.lang.Thread.State: RUNNABLE > > > > at java.net.SocketInputStream.socketRead0(Native Method) > > > > at java.net.SocketInputStream.read(SocketInputStream.java:129) > > > > at > java.io.BufferedInputStream.fill(BufferedInputStream.java:218) > > > > at > java.io.BufferedInputStream.read1(BufferedInputStream.java:256) > > > > at > java.io.BufferedInputStream.read(BufferedInputStream.java:313) > > > > at > org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:607) > > > > at > org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:545) > > > > at > org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:67 > 2) > > > > at > org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.ja > va:876) > > > > at > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.j > ava:684) > > > > at java.lang.Thread.run(Thread.java:595) > > > > > > > > "TP-Processor118" - Thread t...@2293 > > > >java.lang.Thread.State: WAITING on > org.apache.tomcat.util.threads.threadpool$controlrunna...@3579cafe > > > > at java.lang.Object.wait(Native Method) > > > > at java.lang.Object.wait(Object.java:474) > > > > at > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.j > ava:656) > > > > at java.lang.Thread.run(Thread.java:595) > > > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity
Yes, those two look like "waiting for next request on an existing connection from the web server to Tomcat" and "sitting idle in the pool, waiting for a new connection to handle". On 18.05.2009 22:44, Pantvaidya, Vishwajit wrote: >>> [Pantvaidya, Vishwajit] Here are figures from 3 of the servers which > >> have > >>> not yet run out of threads (so the thread count does not add up to 200). > >> I > >>> have taken these late at night when no users are present, so I was > >>> expecting all threads to be Waiting for tomcat thread-pool. > > >>> 1. Total TP-Processor threads 48, Waiting 46, Runnable 2 > >>> 2. Total TP-Processor threads 40, Waiting 29, Runnable 11 > >>> 3. Total TP-Processor threads 120, Waiting 7, Runnable 113 > > > > [Pantvaidya, Vishwajit] Since Rainer mentioned that he would like to see more > of the stack trace, here are the complete stack traces for a Runnable and > Waiting thread from #3 above. All Runnable/Waiting threads from all the above > cases have same stack trace as below: > > > > "TP-Processor119" - Thread t...@2294 > >java.lang.Thread.State: RUNNABLE > > at java.net.SocketInputStream.socketRead0(Native Method) > > at java.net.SocketInputStream.read(SocketInputStream.java:129) > > at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) > > at java.io.BufferedInputStream.read1(BufferedInputStream.java:256) > > at java.io.BufferedInputStream.read(BufferedInputStream.java:313) > > at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:607) > > at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:545) > > at > org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:672) > > at > org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:876) > > at > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684) > > at java.lang.Thread.run(Thread.java:595) > > > > "TP-Processor118" - Thread t...@2293 > >java.lang.Thread.State: WAITING on > org.apache.tomcat.util.threads.threadpool$controlrunna...@3579cafe > > at java.lang.Object.wait(Native Method) > > at java.lang.Object.wait(Object.java:474) > > at > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656) > > at java.lang.Thread.run(Thread.java:595) > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity
> > [Pantvaidya, Vishwajit] Here are figures from 3 of the servers which > have > > not yet run out of threads (so the thread count does not add up to 200). > I > > have taken these late at night when no users are present, so I was > > expecting all threads to be Waiting for tomcat thread-pool. > > > > 1. Total TP-Processor threads 48, Waiting 46, Runnable 2 > > 2. Total TP-Processor threads 40, Waiting 29, Runnable 11 > > 3. Total TP-Processor threads 120, Waiting 7, Runnable 113 > > > > [Pantvaidya, Vishwajit] Since Rainer mentioned that he would like to see more of the stack trace, here are the complete stack traces for a Runnable and Waiting thread from #3 above. All Runnable/Waiting threads from all the above cases have same stack trace as below: "TP-Processor119" - Thread t...@2294 java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read1(BufferedInputStream.java:256) at java.io.BufferedInputStream.read(BufferedInputStream.java:313) at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:607) at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:545) at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:672) at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:876) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684) at java.lang.Thread.run(Thread.java:595) "TP-Processor118" - Thread t...@2293 java.lang.Thread.State: WAITING on org.apache.tomcat.util.threads.threadpool$controlrunna...@3579cafe at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:474) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656) at java.lang.Thread.run(Thread.java:595)
RE: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity
> From: Pantvaidya, Vishwajit [mailto:vpant...@selectica.com] > Subject: RE: Running out of tomcat threads - why many threads in > RUNNABLEstage even with no activity > > [Pantvaidya, Vishwajit] Posting the thread dumps for the above 3 cases, The list usually filters out attachments, as it has done with yours. Either put them on a publicly accessible web site, or right in the text of the e-mail. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity
> [Pantvaidya, Vishwajit] Here are figures from 3 of the servers which have > not yet run out of threads (so the thread count does not add up to 200). I > have taken these late at night when no users are present, so I was > expecting all threads to be Waiting for tomcat thread-pool. > > 1. Total TP-Processor threads 48, Waiting 46, Runnable 2 > 2. Total TP-Processor threads 40, Waiting 29, Runnable 11 > 3. Total TP-Processor threads 120, Waiting 7, Runnable 113 > > > > [Pantvaidya, Vishwajit] Posting the thread dumps for the above 3 cases, since Rainer mentioned that he would like to see more of the stack trace. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Vishwajit, On 5/18/2009 2:01 PM, Pantvaidya, Vishwajit wrote: > Yes, sure - we will upgrade at some point of > time. But since upgrading all our servers will be some work, that may > not happen right away. Upgrading mod_jk is the least painful of all of these, and the most likely to affect you. > Here are figures from 3 of the servers which > have not yet run out of threads (so the thread count does not add up > to 200). I have taken these late at night when no users are present, > so I was expecting all threads to be Waiting for tomcat thread-pool. > > 1. Total TP-Processor threads 48, Waiting 46, Runnable 2 > 2. Total TP-Processor threads 40, Waiting 29, Runnable 11 > 3. Total TP-Processor threads 120, Waiting 7, Runnable 113 Are you sure you aren't seeing any traffic, even that late at night? What if you watch the access logs? Are there requests actively being serviced? > Do you think this could be because of the > application? I was under the impression that there is some tomcat > config parameter that controls this - which was set to 4. No, Tomcat uses precisely 1 thread to handle each incoming HTTP request. If keepalives are used, multiple requests may be handled by the same thread before it is returned to the pool, or different threads may be used to serve different requests from the single connection, but in either case, no more than 1 thread will be used to service a single HTTP request. >>> My workers config is: >>> >>> Worker...type=ajp13 >>> Worker...cachesize=10 >> Are you using the prefork MPM? If so, cachesize should be /1/. > > [Pantvaidya, Vishwajit] Could you please elaborate. What is the > prefork MPM? The MPM is the concurrency strategy employed by Apache httpd. Either you are using the "prefork MPM" which starts multiple httpd processes to handle requests, or you are using the "worker MPM" which starts multiple threads to handle requests. Actually, mod_jk should be able to auto-detect the appropriate cachesize (called connection_pool_size, now), so you shouldn't have to set this. >>> Worker...cache_timeout=600 Worker...socket_keepalive=1 >>> Worker...recycle_timeout=300 >> Are these timeouts necessary? Why not simply let the connections >> stay alive all the time? >> > [Pantvaidya, Vishwajit] Sure we could. But for any production change, > I would have to offer a good enough reason. What was the "good enough reason" to set those timeouts in the first place? >>> Earlier posts related to this issue on the list seem to >>> recommend tweaking: - several timeouts - JkOptions +DisableReuse >> This will require that every incoming HTTP connection opens up a >> new ajp13 connection to Tomcat. Your performance will totally suck >> if you enable this. But if it's the only way for you to get your >> application working properly, then I guess you'll have to do it. I >> suspect you /will not/ have to enable +DisableReuse. >> > [Pantvaidya, Vishwajit] I was seeing earlier posts on this list > mention some disagreement on the performance impact of setting > +DisableReuse. Otherwise I would not even think of this. +DisableReuse will, without a doubt, decrease your performance. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkoRufgACgkQ9CaO5/Lv0PBGHwCfaOGjjv8A1wOAO0CrBKiMbVhM 9MEAoKD8QUbxfqd4h/8YcppkKAt2J1qM =+kJC -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity
On 14.05.2009 20:29, Pantvaidya, Vishwajit wrote: > I set connectionTimeout in server.xml to 60 and now the RUNNABLE > threads go back to WAITING stage after that time. > > But our other servers which are running the same configuration, same > webapp and do not have connectionTimeout set in server.xml, do not > show so many RUNNABLE threads, but more WAITING threads. So looks > like the threads are getting recycled properly there. > > Any idea why could this be? Could it be the OS (all servers run Linux > but I do not know which flavors/versions)? Different web server configuration? Different load specifics? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity
On 13.05.2009 23:28, Pantvaidya, Vishwajit wrote: > My setup is tomcat 5.5.17 + mod_jk 1.2.15 + httpd 2.2.2. I am using > AJP1.3. Every 2-3 days with no major load, tomcat throws the error: > "SEVERE: All threads (200) are currently busy, waiting..." > > I have been monitoring my tomcat TP-Processor thread behavior over > extended time intervals and observe that: - even when there is no > activity on the server, several TP-Processor threads are in RUNNABLE > state while few are in WAITING state - RUNNABLE threads stack trace > shows "java.lang.Thread.State: RUNNABLE at > java.net.SocketInputStream.socketRead0(Native Method)..." We would need to see more of the stack. It's likely that those are connected to the web server, waiting for the next request. > - WAITING thread stack trace shows "java.lang.Thread.State: WAITING > on > org.apache.tomcat.util.threads.threadpool$controlrunna...@53533c55" Likely idle in the pool, availale to handle new connections. > - tomcat adds 4 new TP-Processor threads when a request comes in and > it can find no WAITING threads > > So I conclude that my tomcat is running out of threads due to many > threads being in RUNNABLE state when actually they should be in > WAITING state. Is that happening because of the socket_keepalive in > my workers.properties shown below? Why are threads added in bunches > of 4 - is there any way to configure this? Those socketRead0 threads (Disclaimer: I already said we need more of the stack to be sure) are connected to the web server, waiting for new requests. As long as the new requests come from one of theose web server processes, no new thread is needed to handle them. socket_keepalive is not directly related to that. It tries to workaround a problem, where some component (e.g. firewall) between web server and Tomcat cuts an idle connection without letting the web server and Tomcat know. If you want to free the thread handling the persistent connections, you caqn use the connection pool timeout on the jk side and also the connection pool minimum size (e.g. setting it to 0). On the Tomcat side use connetionTimeout. Be warned, that jk and Tomcat do not use the same time unit for those parameters. Have a look at the timeouts documentation of mod_jk. > My workers config is: > > Worker...type=ajp13 Worker...cachesize=10 Worker...cache_timeout=600 > Worker...socket_keepalive=1 Worker...recycle_timeout=300 > > Earlier posts related to this issue on the list seem to recommend > tweaking: - several timeouts - JkOptions +DisableReuse Very last resort. Should not be needed and might obscure some other problem. > I am planning to do the following to resolve our problem: - upgrade > jk to latest version - e.g. 1.2.28 - replace recycle_timeout with > connection_pool_timeout - add connectionTimeout in server.xml - add > JkOptions +DisableReuse > > Please let me know if this is okay or suggestions if any. I suspect, that during the time of the "all threads are currently busy" message, something in or behind your app was slow and so requests got stuck in front of Tomcat, the web server pool was growing until it connected 200 web server processes/threads trying to send requests to Tomcat. To find out what the root cause was, you'll need to make the thread dumps during the problem time. Also note, that the maximum concurrency in your web server layer should be a good fit to the maximum concurrency (thread pool size) in the Tomcat layer. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity
Hi Chris, Thanks for your reply. > > On 5/13/2009 5:28 PM, Pantvaidya, Vishwajit wrote: > > My setup is tomcat 5.5.17 + mod_jk 1.2.15 + httpd 2.2.2. I am using > > AJP1.3. > > Old versions of everything. Consider upgrading? > [Pantvaidya, Vishwajit] Yes, sure - we will upgrade at some point of time. But since upgrading all our servers will be some work, that may not happen right away. > > Every 2-3 days with no major load, tomcat throws the error: "SEVERE: > > All threads (200) are currently busy, waiting..." > > > > I have been monitoring my tomcat TP-Processor thread behavior over > > extended time intervals and observe that: - even when there is no > > activity on the server, several TP-Processor threads are in RUNNABLE > > state while few are in WAITING state > > It appears that you have 200 threads available. How many (on average) > are RUNNABLE versus WAITING? (The two counts should add to 200, unless > there's some other state (BLOCKED?) that the threads can be in, but you > didn't mention any other states). [Pantvaidya, Vishwajit] Here are figures from 3 of the servers which have not yet run out of threads (so the thread count does not add up to 200). I have taken these late at night when no users are present, so I was expecting all threads to be Waiting for tomcat thread-pool. 1. Total TP-Processor threads 48, Waiting 46, Runnable 2 2. Total TP-Processor threads 40, Waiting 29, Runnable 11 3. Total TP-Processor threads 120, Waiting 7, Runnable 113 > > > - RUNNABLE threads stack trace shows "java.lang.Thread.State: > > RUNNABLE at java.net.SocketInputStream.socketRead0(Native > > Method)..." > > This indicates that the client has not yet disconnected, and is probably > still sending data. If there were not any data waiting, the state should > be BLOCKED. > > > - WAITING thread stack trace shows "java.lang.Thread.State: WAITING > > on > > org.apache.tomcat.util.threads.threadpool$controlrunna...@53533c55" > > These are idle threads. > > > - tomcat adds 4 new TP-Processor threads when a request comes in and > > it can find no WAITING threads > > Wow, 4 new threads? That seems like 3 too many... [Pantvaidya, Vishwajit] Do you think this could be because of the application? I was under the impression that there is some tomcat config parameter that controls this - which was set to 4. > > > So I conclude that my tomcat is running out of threads due to many > > threads being in RUNNABLE state when actually they should be in > > WAITING state. Is that happening because of the socket_keepalive in > > my workers.properties shown below? > > worker.socket_keepalive just keeps the connection between Apache httpd > and Tomcat "alive" in case you have an overzealous firewall that closes > inactive connections. The request processor shouldn't be affected by > this setting. > > > Why are threads added in bunches of 4 - is there any way to configure > > this? > > > > My workers config is: > > > > Worker...type=ajp13 Worker...cachesize=10 > > Are you using the prefork MPM? If so, cachesize should be /1/. > [Pantvaidya, Vishwajit] Could you please elaborate. What is the prefork MPM? > > Worker...cache_timeout=600 Worker...socket_keepalive=1 > > Worker...recycle_timeout=300 > > Are these timeouts necessary? Why not simply let the connections stay > alive all the time? > [Pantvaidya, Vishwajit] Sure we could. But for any production change, I would have to offer a good enough reason. > > Earlier posts related to this issue on the list seem to recommend > > tweaking: - several timeouts - JkOptions +DisableReuse > > This will require that every incoming HTTP connection opens up a new > ajp13 connection to Tomcat. Your performance will totally suck if you > enable this. But if it's the only way for you to get your application > working properly, then I guess you'll have to do it. I suspect you /will > not/ have to enable +DisableReuse. > [Pantvaidya, Vishwajit] I was seeing earlier posts on this list mention some disagreement on the performance impact of setting +DisableReuse. Otherwise I would not even think of this. By the way, the above 3 figures I provided are without connectiontimeout being set for Connector element in server.xml. - Vish. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Vishwajit, On 5/13/2009 5:28 PM, Pantvaidya, Vishwajit wrote: > My setup is tomcat 5.5.17 + mod_jk 1.2.15 + httpd 2.2.2. I am using > AJP1.3. Old versions of everything. Consider upgrading? > Every 2-3 days with no major load, tomcat throws the error: "SEVERE: > All threads (200) are currently busy, waiting..." > > I have been monitoring my tomcat TP-Processor thread behavior over > extended time intervals and observe that: - even when there is no > activity on the server, several TP-Processor threads are in RUNNABLE > state while few are in WAITING state It appears that you have 200 threads available. How many (on average) are RUNNABLE versus WAITING? (The two counts should add to 200, unless there's some other state (BLOCKED?) that the threads can be in, but you didn't mention any other states). > - RUNNABLE threads stack trace shows "java.lang.Thread.State: > RUNNABLE at java.net.SocketInputStream.socketRead0(Native > Method)..." This indicates that the client has not yet disconnected, and is probably still sending data. If there were not any data waiting, the state should be BLOCKED. > - WAITING thread stack trace shows "java.lang.Thread.State: WAITING > on > org.apache.tomcat.util.threads.threadpool$controlrunna...@53533c55" These are idle threads. > - tomcat adds 4 new TP-Processor threads when a request comes in and > it can find no WAITING threads Wow, 4 new threads? That seems like 3 too many... > So I conclude that my tomcat is running out of threads due to many > threads being in RUNNABLE state when actually they should be in > WAITING state. Is that happening because of the socket_keepalive in > my workers.properties shown below? worker.socket_keepalive just keeps the connection between Apache httpd and Tomcat "alive" in case you have an overzealous firewall that closes inactive connections. The request processor shouldn't be affected by this setting. > Why are threads added in bunches of 4 - is there any way to configure > this? > > My workers config is: > > Worker...type=ajp13 Worker...cachesize=10 Are you using the prefork MPM? If so, cachesize should be /1/. > Worker...cache_timeout=600 Worker...socket_keepalive=1 > Worker...recycle_timeout=300 Are these timeouts necessary? Why not simply let the connections stay alive all the time? > Earlier posts related to this issue on the list seem to recommend > tweaking: - several timeouts - JkOptions +DisableReuse This will require that every incoming HTTP connection opens up a new ajp13 connection to Tomcat. Your performance will totally suck if you enable this. But if it's the only way for you to get your application working properly, then I guess you'll have to do it. I suspect you /will not/ have to enable +DisableReuse. > I am planning to do the following to resolve our problem: - upgrade > jk to latest version - e.g. 1.2.28 Upgrading is (almost) always a good idea. > - replace recycle_timeout with connection_pool_timeout - add > connectionTimeout in server.xml - add JkOptions +DisableReuse I think these settings will only reduce performance. If you want my advice, I'd simplify your configuration as much as possible, then add settings as you need them. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkoNf5EACgkQ9CaO5/Lv0PCswQCfQ101OnbCFnTEOu0e8wqlVt1Q gycAn1hohlsIrYyEEno0jGrGglE2yInF =hWV8 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity
> From: Pantvaidya, Vishwajit [mailto:vpant...@selectica.com] > Subject: RE: Running out of tomcat threads - why many threads in > RUNNABLEstage even with no activity > > Since I did not get any responses to this, just wanted to ask - did I > post this to the wrong list and should I be posting this to the tomcat > developers list instead? This should be the correct list, but there's probably only one person who can definitively answer your question and he may be busy (or on holiday). - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
