RE: Tomcat maxThreads Issue
Update, I finally got my vendor on the line. He is telling me that this entry in my server.xml: connector port=81 protocol=HTTP/1.1 Server- maxThreads=600 connectionTimeout=2 redirectpPort=443 / Is the reason I'm seeing this in my catalina.out. He is stating that If I put an address (the same address Of the linux server that the tomcat instance is running on) that that message will go away and the maxThreads will actually move to 600. So it should look like where 10.10.10.10 is the internal IP of the tomcat app server connector port=81 protocol=HTTP/1.1 Server10.10.10.10 maxThreads=600 connectionTimeout=2 redirectpPort=443 / Anyone agree? -Original Message- From: Caldarale, Charles R [mailto:chuck.caldar...@unisys.com] Sent: Wednesday, June 10, 2009 3:25 PM To: Tomcat Users List Subject: RE: Tomcat maxThreads Issue From: Jones, Keven [mailto:keven.jo...@ncr.com] Subject: RE: Tomcat maxThreads Issue Connector port=8011 enableLookups=false protocol=AJP/1.3 redirectPort=8443 maxThreads=400 connectionTimeout=2/ The syntax is ok, but I'd have to look at the code to see if connectionTimeout applies to anything other than the first request coming through. (AJP connections are intended to be persistent.) - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat maxThreads Issue
From: Jones, Keven [mailto:keven.jo...@ncr.com] Subject: RE: Tomcat maxThreads Issue He is stating that If I put an address (the same address Of the linux server that the tomcat instance is running on) that that message will go away and the maxThreads will actually move to 600. Anyone agree? Nope - your vendor is sadly mistaken or confused (or you misinterpreted what he said). The address (not server) attribute in a Connector will limit the listen to just that IP address, whereas not including an address attribute results in listening on all IP addresses configured for the TCP/IP stack, including 10.10.10.10. The maxThreads attribute applies either way. Your vendor is also confused, because that's not the Connector typically used between Tomcat and httpd; it's normally the AJP one, if you're using mod_jk or mod_proxy_ajp. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat maxThreads Issue
On 10.06.2009 21:25, Caldarale, Charles R wrote: From: Jones, Keven [mailto:keven.jo...@ncr.com] Subject: RE: Tomcat maxThreads Issue Connector port=8011 enableLookups=false protocol=AJP/1.3 redirectPort=8443 maxThreads=400 connectionTimeout=2/ The syntax is ok, but I'd have to look at the code to see if connectionTimeout applies to anything other than the first request coming through. (AJP connections are intended to be persistent.) Haven't read through all of the thread yet, but yes, connectionTimeout on the AJP connector also closes the connection if the time between two requests on an existing connections is longer than connectionTimeout. 2 (20 seconds) is relatively short though. Setting it to something like 30 should be short enough. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat maxThreads Issue
On 10.06.2009 18:26, Jones, Keven wrote: FYI, I'm just the linux admin as the applicatons group/developer and coder is not making any headway into resolving this. This is a mobility application (ie. Mobile banking, so you go to your iphone and go to your bank.mobi..)not that it matter but Ok, So yesterday I did perform a kill -3 and got a THREAD DUMP in the catalina.out file. I have copied most of it below: The dump during times where everything is fine doesn't reveal much. You (unfortunately) need to do this, when the problem occurs. If you manage to catch such an event, do 3 thread dumps, each a few seconds apart from the next. That way one can also decide, whether a situation is static or only short term. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat maxThreads Issue
This is a thread dump during a problem. I will take your advice and execute 3 whan the next event happens. Thank you for your help! -Original Message- From: Rainer Jung rainer.j...@kippdata.de Sent: Thursday, June 11, 2009 6:08 PM To: Tomcat Users List users@tomcat.apache.org Subject: Re: Tomcat maxThreads Issue On 10.06.2009 18:26, Jones, Keven wrote: FYI, I'm just the linux admin as the applicatons group/developer and coder is not making any headway into resolving this. This is a mobility application (ie. Mobile banking, so you go to your iphone and go to your bank.mobi..)not that it matter but Ok, So yesterday I did perform a kill -3 and got a THREAD DUMP in the catalina.out file. I have copied most of it below: The dump during times where everything is fine doesn't reveal much. You (unfortunately) need to do this, when the problem occurs. If you manage to catch such an event, do 3 thread dumps, each a few seconds apart from the next. That way one can also decide, whether a situation is static or only short term. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat maxThreads Issue
On 12.06.2009 00:13, Jones, Keven wrote: This is a thread dump during a problem. I will take your advice and execute 3 whan the next event happens. Thank you for your help! Then your problem doesn't last long enough. The presented dump doesn't contain stacks for 200 connector threads of one connector, so the problem (exhausted connector thread pool) was already over. Regards, Rainer -Original Message- From: Rainer Jung rainer.j...@kippdata.de Sent: Thursday, June 11, 2009 6:08 PM To: Tomcat Users List users@tomcat.apache.org Subject: Re: Tomcat maxThreads Issue On 10.06.2009 18:26, Jones, Keven wrote: FYI, I'm just the linux admin as the applicatons group/developer and coder is not making any headway into resolving this. This is a mobility application (ie. Mobile banking, so you go to your iphone and go to your bank.mobi..)not that it matter but Ok, So yesterday I did perform a kill -3 and got a THREAD DUMP in the catalina.out file. I have copied most of it below: The dump during times where everything is fine doesn't reveal much. You (unfortunately) need to do this, when the problem occurs. If you manage to catch such an event, do 3 thread dumps, each a few seconds apart from the next. That way one can also decide, whether a situation is static or only short term. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat maxThreads Issue
Jones, Keven wrote: Hello, I have Tomcat (can't figure out what version as the version.sh script is giving all kinds of errors) running on separate Linux servers. So an overview of my config is 1 web server, 4 app servers (tomcat), and 1 db server. We were seeing an increase in the httpd processes on the web server to the point where I had to increase the MaxClients on the apache 2 server to 2048. I finally figured out that the normal number of httpd daemons on the web server (50-100) was spiking due to tomcat processes stalling or going completely down. So I started looking at why tomcat was failing and I found this error message in the ACStarError.log : SEVERE: All threads (200) are currently busy, waiting. Increase maxThreads (200) or check the servlet status. So I checked the server.xml file and found the following: Connector port=81 protocol=HTTP/1.1 server=- maxThreads=600 connectionTimeout=2 redirectPort=443 / !-- Awele 20070402 For SSL Requests, processed non-SSL, due to Coyote SSL Accelerator -- !-- Connector port=443 address=10.5.3.9 protocol=HTTP/1.1 server=- secure=true maxThreads=600 connectionTimeout=2 / -- !-- Define a SSL HTTP/1.1 Connector on port 8443 This connector uses the JSSE configuration, when using APR, the connector should be using the OpenSSL style configuration described in the APR documentation -- !-- Connector port=8443 protocol=HTTP/1.1 SSLEnabled=true maxThreads=150 scheme=https secure=true clientAuth=false sslProtocol=TLS / -- So It looks like the maxThreads has already been increased to 600. The server has been rebooted and tomcat stopped/started multiple times. Again all servers are Red Hat 5.2. Is there an additional parameter that needs to be set in order for tomcat to actually increase the maxThreads ? Thank You That is the http connector. What about the APR connector? You can leave out stuff that is commented out. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat maxThreads Issue
Here is the complete config file. Thanks for the quick reply. -Original Message- From: Mark Thomas [mailto:ma...@apache.org] Sent: Wednesday, June 10, 2009 11:14 AM To: Tomcat Users List Subject: Re: Tomcat maxThreads Issue Jones, Keven wrote: Hello, I have Tomcat (can't figure out what version as the version.sh script is giving all kinds of errors) running on separate Linux servers. So an overview of my config is 1 web server, 4 app servers (tomcat), and 1 db server. We were seeing an increase in the httpd processes on the web server to the point where I had to increase the MaxClients on the apache 2 server to 2048. I finally figured out that the normal number of httpd daemons on the web server (50-100) was spiking due to tomcat processes stalling or going completely down. So I started looking at why tomcat was failing and I found this error message in the ACStarError.log : SEVERE: All threads (200) are currently busy, waiting. Increase maxThreads (200) or check the servlet status. So I checked the server.xml file and found the following: Connector port=81 protocol=HTTP/1.1 server=- maxThreads=600 connectionTimeout=2 redirectPort=443 / !-- Awele 20070402 For SSL Requests, processed non-SSL, due to Coyote SSL Accelerator -- !-- Connector port=443 address=10.5.3.9 protocol=HTTP/1.1 server=- secure=true maxThreads=600 connectionTimeout=2 / -- !-- Define a SSL HTTP/1.1 Connector on port 8443 This connector uses the JSSE configuration, when using APR, the connector should be using the OpenSSL style configuration described in the APR documentation -- !-- Connector port=8443 protocol=HTTP/1.1 SSLEnabled=true maxThreads=150 scheme=https secure=true clientAuth=false sslProtocol=TLS / -- So It looks like the maxThreads has already been increased to 600. The server has been rebooted and tomcat stopped/started multiple times. Again all servers are Red Hat 5.2. Is there an additional parameter that needs to be set in order for tomcat to actually increase the maxThreads ? Thank You That is the http connector. What about the APR connector? You can leave out stuff that is commented out. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat maxThreads Issue
From: Jones, Keven [mailto:keven.jo...@ncr.com] Here is the complete config file. Thanks for the quick reply. The list strips attachments - please paste inline. - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat maxThreads Issue
processes all example.
Documentation at: /docs/config/valve.html --
!--
Valve className=org.apache.catalina.valves.AccessLogValve
directory=logs
prefix=localhost_access_log. suffix=.txt pattern=%h %l %u
%t quot;%rquot; %s %b quot;%{Referer}iquot;
quot;%{User-Agent}iquot; %{Host}i %p resolveHosts=false/
--
/Host
/Engine
/Service
/Server
-Original Message-
From: Peter Crowther [mailto:peter.crowt...@melandra.com]
Sent: Wednesday, June 10, 2009 11:19 AM
To: 'Tomcat Users List'
Subject: RE: Tomcat maxThreads Issue
From: Jones, Keven [mailto:keven.jo...@ncr.com] Here is the complete
config file. Thanks for the quick reply.
The list strips attachments - please paste inline.
- Peter
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat maxThreads Issue
From: Jones, Keven [mailto:keven.jo...@ncr.com] Subject: RE: Tomcat maxThreads Issue !-- Define an AJP 1.3 Connector on port 8009 -- Connector port=8011 enableLookups=false protocol=AJP/1.3 redirectPort=8443 / This is the connector Mark was referring to (he should have said AJP, not APR). Since you have no maxThreads attribute, it defaults to 200, which may or may not be appropriate for your expected load. The real question is what are those 200 threads doing? Try taking a couple of thread dumps when you get into the problem and see what they're stuck on. Engine name=Standalone defaultHost=localhost jvmRoute=mweb3a You really should have the Engine and Service names the same, to keep JMX happy. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat maxThreads Issue
I don't have any tomcat knowledge. Thank you for the reply. How do I A. get thread dumps when the processes are dying And B. You mentioned the engine and service name. I see we have Engine set to Standalone, And I see the Service is catalina (?) which should I adjust? Thx again -Original Message- From: Caldarale, Charles R [mailto:chuck.caldar...@unisys.com] Sent: Wednesday, June 10, 2009 11:33 AM To: Tomcat Users List Subject: RE: Tomcat maxThreads Issue From: Jones, Keven [mailto:keven.jo...@ncr.com] Subject: RE: Tomcat maxThreads Issue !-- Define an AJP 1.3 Connector on port 8009 -- Connector port=8011 enableLookups=false protocol=AJP/1.3 redirectPort=8443 / This is the connector Mark was referring to (he should have said AJP, not APR). Since you have no maxThreads attribute, it defaults to 200, which may or may not be appropriate for your expected load. The real question is what are those 200 threads doing? Try taking a couple of thread dumps when you get into the problem and see what they're stuck on. Engine name=Standalone defaultHost=localhost jvmRoute=mweb3a You really should have the Engine and Service names the same, to keep JMX happy. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat maxThreads Issue
From: Jones, Keven [mailto:keven.jo...@ncr.com] How do I A. get thread dumps when the processes are dying The top of http://java.sun.com/developer/technicalArticles/Programming/Stacktrace/ is good general reading. kill -QUIT pid of Java process If ps is confused by all the threads, depending on your Java version, you might be able to use jps (http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jps.html) to get the pid. Depending on your Java version, jstack (http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jstack.html) will also give you thread dumps. - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat maxThreads Issue
/apache/tomcat/util/threads/ThreadPool$ControlRunnable.run(ThreadPool.java:658) ^-- Lock released while waiting: org/apache/tomcat/util/threads/threadpool$controlrunna...@0x19 812ae0[fat lock] at java/lang/Thread.run(Thread.java:595)[optimized] at jrockit/vm/RNI.c2java()V(Native Method) -- end of trace aRequest: -Completed- id=49508 idx=0x196 tid=22091 prio=5 alive, in native, waiting, daemon -- Waiting for notification on: org/apache/tomcat/util/threads/threadpool$controlrunna...@0x198 73a80[fat lock] at jrockit/vm/Threads.waitForSignal(J)Z(Native Method) at jrockit/vm/Locks.wait(Ljava/lang/Object;J)V(Unknown Source)[inlined] at java/lang/Object.wait()V(Native Method)[optimized] at org/apache/tomcat/util/threads/ThreadPool$ControlRunnable.run(ThreadPool.java:658) ^-- Lock released while waiting: org/apache/tomcat/util/threads/threadpool$controlrunna...@0x19 873a80[fat lock] at java/lang/Thread.run(Thread.java:595)[optimized] at jrockit/vm/RNI.c2java()V(Native Method) -- end of trace = END OF THREAD DUMP === -Original Message- From: Peter Crowther [mailto:peter.crowt...@melandra.com] Sent: Wednesday, June 10, 2009 12:16 PM To: 'Tomcat Users List' Subject: RE: Tomcat maxThreads Issue From: Jones, Keven [mailto:keven.jo...@ncr.com] How do I A. get thread dumps when the processes are dying The top of http://java.sun.com/developer/technicalArticles/Programming/Stacktrace/ is good general reading. kill -QUIT pid of Java process If ps is confused by all the threads, depending on your Java version, you might be able to use jps (http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jps.html) to get the pid. Depending on your Java version, jstack (http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jstack.html) will also give you thread dumps. - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat maxThreads Issue
OK, so that's a JRockit JVM, not a Sun JVM. Remind me not to point you at any more Sun documentation! As a rough look at that, there are a few threads in the HTTP connector thread pool, a bunch of threads reading data from JK connections, and one or two in your app code. I've never looked at the innards of JK, so will have to leave that to one of the JK experts to interpret. - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat maxThreads Issue
Looking at the history of the thread - here are some guesses You have a connector on port 81 (http) and a connector on 8011 (ajp) and commented out connector for 443. The port 81 connector and commented out connector are set to 600 - which means absolutely nothing if apache is talking to tomcat via jk. Set |maxThreads on the connector for using port 8011 and life should be better. If there are mulitple apache servers in front of tomcat - you need to ensure you have (#apaches * max number connections per apache) threads available on each tomcat. The thread history seems to indicate you only have one apache with 100 workers max. So raising the maxThreads in tomcat from 200 to something higher is not needed. Is there a firewall between apache and tomcat? If so - it could mean that your firewall is timing out the connection and then bad things happen. | http://tomcat.apache.org/tomcat-6.0-doc/config/ajp.html -Tim - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat maxThreads Issue
I encountered a similar issue of numerous Httpd processes spawning b/c of Tomcat. The issue for me was that Tomcat was running out of memory, and Httpd for some reason thought spawning a bunch a threads would help ;) Do you know how much memory these instances of Tomcat are using? The default JVM memory is ridiculously low and requires the adjustment to higher maximum value for production servers. Grep heap catalina.out to see if tomcat is indeed running out of memory. On Wed, Jun 10, 2009 at 12:44 PM, Peter Crowther peter.crowt...@melandra.com wrote: OK, so that's a JRockit JVM, not a Sun JVM. Remind me not to point you at any more Sun documentation! As a rough look at that, there are a few threads in the HTTP connector thread pool, a bunch of threads reading data from JK connections, and one or two in your app code. I've never looked at the innards of JK, so will have to leave that to one of the JK experts to interpret. - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat maxThreads Issue
I did a grep for heap in catalina.out but do not see any out of memory entries. How do I check the default JVM memory setting and how do I adjust it? Thx -Original Message- From: Jonathan Mast [mailto:jhmast.develo...@gmail.com] Sent: Wednesday, June 10, 2009 1:04 PM To: Tomcat Users List Subject: Re: Tomcat maxThreads Issue I encountered a similar issue of numerous Httpd processes spawning b/c of Tomcat. The issue for me was that Tomcat was running out of memory, and Httpd for some reason thought spawning a bunch a threads would help ;) Do you know how much memory these instances of Tomcat are using? The default JVM memory is ridiculously low and requires the adjustment to higher maximum value for production servers. Grep heap catalina.out to see if tomcat is indeed running out of memory. On Wed, Jun 10, 2009 at 12:44 PM, Peter Crowther peter.crowt...@melandra.com wrote: OK, so that's a JRockit JVM, not a Sun JVM. Remind me not to point you at any more Sun documentation! As a rough look at that, there are a few threads in the HTTP connector thread pool, a bunch of threads reading data from JK connections, and one or two in your app code. I've never looked at the innards of JK, so will have to leave that to one of the JK experts to interpret. - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat maxThreads Issue
Thank you Tim I am going to attempt to set the maxThreads on 8011 to 600 and hope it helps. The config is 1 linux apache2 server with 4 linux tomcat servers and each tomcat server has 4 sepearte tomcat instances. So we have a total of 16 tomcat instances across 4 servers. Yes, there is a firewall between the web sever and the app(tomcat)/db(mysql) server. KJ -Original Message- From: Tim Funk [mailto:funk...@apache.org] Sent: Wednesday, June 10, 2009 1:02 PM To: Tomcat Users List Subject: Re: Tomcat maxThreads Issue Looking at the history of the thread - here are some guesses You have a connector on port 81 (http) and a connector on 8011 (ajp) and commented out connector for 443. The port 81 connector and commented out connector are set to 600 - which means absolutely nothing if apache is talking to tomcat via jk. Set |maxThreads on the connector for using port 8011 and life should be better. If there are mulitple apache servers in front of tomcat - you need to ensure you have (#apaches * max number connections per apache) threads available on each tomcat. The thread history seems to indicate you only have one apache with 100 workers max. So raising the maxThreads in tomcat from 200 to something higher is not needed. Is there a firewall between apache and tomcat? If so - it could mean that your firewall is timing out the connection and then bad things happen. | http://tomcat.apache.org/tomcat-6.0-doc/config/ajp.html -Tim - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat maxThreads Issue
If your not scared of the performance penalty (which these days isn't scary unless you really have high load) - try adding this to your apache config - it will end the connection between apache and tomcat after each request. This virtually eliminates the chance of a firewall dropping connections for you. JkOptions +DisableReuse YOu prob dont want this permanantly but its a good test http://tomcat.apache.org/connectors-doc/reference/apache.html -Tim Jones, Keven wrote: Thank you Tim I am going to attempt to set the maxThreads on 8011 to 600 and hope it helps. The config is 1 linux apache2 server with 4 linux tomcat servers and each tomcat server has 4 sepearte tomcat instances. So we have a total of 16 tomcat instances across 4 servers. Yes, there is a firewall between the web sever and the app(tomcat)/db(mysql) server. KJ -Original Message- From: Tim Funk [mailto:funk...@apache.org] Sent: Wednesday, June 10, 2009 1:02 PM To: Tomcat Users List Subject: Re: Tomcat maxThreads Issue Looking at the history of the thread - here are some guesses You have a connector on port 81 (http) and a connector on 8011 (ajp) and commented out connector for 443. The port 81 connector and commented out connector are set to 600 - which means absolutely nothing if apache is talking to tomcat via jk. Set |maxThreads on the connector for using port 8011 and life should be better. If there are mulitple apache servers in front of tomcat - you need to ensure you have (#apaches * max number connections per apache) threads available on each tomcat. The thread history seems to indicate you only have one apache with 100 workers max. So raising the maxThreads in tomcat from 200 to something higher is not needed. Is there a firewall between apache and tomcat? If so - it could mean that your firewall is timing out the connection and then bad things happen. | http://tomcat.apache.org/tomcat-6.0-doc/config/ajp.html -Tim - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat maxThreads Issue
From: Jones, Keven [mailto:keven.jo...@ncr.com] Subject: RE: Tomcat maxThreads Issue I am going to attempt to set the maxThreads on 8011 to 600 and hope it helps. It will likely only postpone the problem. Yes, there is a firewall between the web sever and the app(tomcat)/db(mysql) server. The question was whether or not there's a firewall between httpd and Tomcat, not Tomcat and the DB server. If there is a firewall between httpd and Tomcat, it could be silently dropping the AJP connections, leaving Tomcat threads stranded on unusable ports. There has been much discussion about this of late on this mailing list; I'll try to find some pointers for you. Unless you're seeing OutOfMemoryErrors in your Tomcat logs, the heap sizing is very likely a red herring. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat maxThreads Issue
Well I'm not sure how to check the current memory settings but I sure know how to adjust them :) Presuming that you downloaded Tomcat directly and installed manually: open $TOMCAT/bin/catalina.sh Add the following somewhere near the top (I place mine after all the explanatory comments): JAVA_OPTS= -Xmx1024m This is tells java to allow a maximum of 1024 megabytes of memory to this instance of tomcat. Adjust appropriately. Please note: * the leading SPACE before the -Xmx. * this is not a labeled parameter, therefore no = btwn the option and value. On Wed, Jun 10, 2009 at 1:12 PM, Jones, Keven keven.jo...@ncr.com wrote: Thank you Tim I am going to attempt to set the maxThreads on 8011 to 600 and hope it helps. The config is 1 linux apache2 server with 4 linux tomcat servers and each tomcat server has 4 sepearte tomcat instances. So we have a total of 16 tomcat instances across 4 servers. Yes, there is a firewall between the web sever and the app(tomcat)/db(mysql) server. KJ -Original Message- From: Tim Funk [mailto:funk...@apache.org] Sent: Wednesday, June 10, 2009 1:02 PM To: Tomcat Users List Subject: Re: Tomcat maxThreads Issue Looking at the history of the thread - here are some guesses You have a connector on port 81 (http) and a connector on 8011 (ajp) and commented out connector for 443. The port 81 connector and commented out connector are set to 600 - which means absolutely nothing if apache is talking to tomcat via jk. Set |maxThreads on the connector for using port 8011 and life should be better. If there are mulitple apache servers in front of tomcat - you need to ensure you have (#apaches * max number connections per apache) threads available on each tomcat. The thread history seems to indicate you only have one apache with 100 workers max. So raising the maxThreads in tomcat from 200 to something higher is not needed. Is there a firewall between apache and tomcat? If so - it could mean that your firewall is timing out the connection and then bad things happen. | http://tomcat.apache.org/tomcat-6.0-doc/config/ajp.html -Tim - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat maxThreads Issue
Cool, Also, I'm examining my tomcat server.xml conf file for each instnace on each server. I am finding that Tomcat1 AJP connector is 8011, Tomcat2 AJP 8012, Tomcat3 AJP 8013, and Tomcat4 AJP 8014. So I will try and Add a maxThreads 600 on each of these. Thanks -Original Message- From: Tim Funk [mailto:funk...@apache.org] Sent: Wednesday, June 10, 2009 1:33 PM To: Tomcat Users List Subject: Re: Tomcat maxThreads Issue If your not scared of the performance penalty (which these days isn't scary unless you really have high load) - try adding this to your apache config - it will end the connection between apache and tomcat after each request. This virtually eliminates the chance of a firewall dropping connections for you. JkOptions +DisableReuse YOu prob dont want this permanantly but its a good test http://tomcat.apache.org/connectors-doc/reference/apache.html -Tim Jones, Keven wrote: Thank you Tim I am going to attempt to set the maxThreads on 8011 to 600 and hope it helps. The config is 1 linux apache2 server with 4 linux tomcat servers and each tomcat server has 4 sepearte tomcat instances. So we have a total of 16 tomcat instances across 4 servers. Yes, there is a firewall between the web sever and the app(tomcat)/db(mysql) server. KJ -Original Message- From: Tim Funk [mailto:funk...@apache.org] Sent: Wednesday, June 10, 2009 1:02 PM To: Tomcat Users List Subject: Re: Tomcat maxThreads Issue Looking at the history of the thread - here are some guesses You have a connector on port 81 (http) and a connector on 8011 (ajp) and commented out connector for 443. The port 81 connector and commented out connector are set to 600 - which means absolutely nothing if apache is talking to tomcat via jk. Set |maxThreads on the connector for using port 8011 and life should be better. If there are mulitple apache servers in front of tomcat - you need to ensure you have (#apaches * max number connections per apache) threads available on each tomcat. The thread history seems to indicate you only have one apache with 100 workers max. So raising the maxThreads in tomcat from 200 to something higher is not needed. Is there a firewall between apache and tomcat? If so - it could mean that your firewall is timing out the connection and then bad things happen. | http://tomcat.apache.org/tomcat-6.0-doc/config/ajp.html -Tim - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat maxThreads Issue
From: Jonathan Mast [mailto:jhmast.develo...@gmail.com] Subject: Re: Tomcat maxThreads Issue Add the following somewhere near the top (I place mine after all the explanatory comments): JAVA_OPTS= -Xmx1024m Again, if there's no evidence of a heap issue (and none has been presented) *do not* change the heap settings - at best that can only muddy the water. If you want to look at garbage collection activity, use -verbose:gc (which should work even on JRockit) to get information about how often a GC occurs. Unless it's happening very frequently, leave the heap settings alone. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat maxThreads Issue
From: Jones, Keven [mailto:keven.jo...@ncr.com] Subject: RE: Tomcat maxThreads Issue I see we have Engine set to Standalone, And I see the Service is catalina (?) I would leave the name as Catalina (it's case sensitive), since that's what everyone expects (unless you have multiple Service elements - a very rare need). - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat maxThreads Issue
From: Caldarale, Charles R Subject: RE: Tomcat maxThreads Issue There has been much discussion about this of late on this mailing list; I'll try to find some pointers for you. This is a useful reference: http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html#Firewall%20Connection%20Dropping The best (and unfortunately, biggest) discussion of this issue is here: http://marc.info/?t=12422502151r=1w=2 The most useful messages in this 61-posting thread are Rainer's, should you want to browse through them. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat maxThreads Issue
Does the syntax of this look correct to everyone? I have added maxThreads=400 and connectionTimeout=2(not sure if I need the connectionTimeout) Connector port=8011 enableLookups=false protocol=AJP/1.3 redirectPort=8443 maxThreads=400 connectionTimeout=2/ Thanks everyone -Original Message- From: Tim Funk [mailto:funk...@apache.org] Sent: Wednesday, June 10, 2009 1:02 PM To: Tomcat Users List Subject: Re: Tomcat maxThreads Issue Looking at the history of the thread - here are some guesses You have a connector on port 81 (http) and a connector on 8011 (ajp) and commented out connector for 443. The port 81 connector and commented out connector are set to 600 - which means absolutely nothing if apache is talking to tomcat via jk. Set |maxThreads on the connector for using port 8011 and life should be better. If there are mulitple apache servers in front of tomcat - you need to ensure you have (#apaches * max number connections per apache) threads available on each tomcat. The thread history seems to indicate you only have one apache with 100 workers max. So raising the maxThreads in tomcat from 200 to something higher is not needed. Is there a firewall between apache and tomcat? If so - it could mean that your firewall is timing out the connection and then bad things happen. | http://tomcat.apache.org/tomcat-6.0-doc/config/ajp.html -Tim - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat maxThreads Issue
From: Jones, Keven [mailto:keven.jo...@ncr.com] Subject: RE: Tomcat maxThreads Issue Connector port=8011 enableLookups=false protocol=AJP/1.3 redirectPort=8443 maxThreads=400 connectionTimeout=2/ The syntax is ok, but I'd have to look at the code to see if connectionTimeout applies to anything other than the first request coming through. (AJP connections are intended to be persistent.) - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat maxThreads Issue
What does your Apache access_log say at the time when tomcat threads are exhausted ? I had a very similar issue in my environment. The culprit turned out to be JkMount options. Apache was sending all requests, whether it is for static or dynamic content to tomcat. At the same time my tomcats locked up, I was seeing a lot of requests for static content(gifs in my case) in the apache access_log Setting up mod_jk to only send *.jsp to tomcats fixed the issue for me. From: Jones, Keven keven.jo...@ncr.com To: users@tomcat.apache.org users@tomcat.apache.org Sent: Wednesday, June 10, 2009 11:11:11 AM Subject: Tomcat maxThreads Issue Hello, I have Tomcat (can't figure out what version as the version.sh script is giving all kinds of errors) running on separate Linux servers. So an overview of my config is 1 web server, 4 app servers (tomcat), and 1 db server. We were seeing an increase in the httpd processes on the web server to the point where I had to increase the MaxClients on the apache 2 server to 2048. I finally figured out that the normal number of httpd daemons on the web server (50-100) was spiking due to tomcat processes stalling or going completely down. So I started looking at why tomcat was failing and I found this error message in the ACStarError.log : SEVERE: All threads (200) are currently busy, waiting. Increase maxThreads (200) or check the servlet status. So I checked the server.xml file and found the following: Connector port=81 protocol=HTTP/1.1 server=- maxThreads=600 connectionTimeout=2 redirectPort=443 / !-- Awele 20070402 For SSL Requests, processed non-SSL, due to Coyote SSL Accelerator -- !-- Connector port=443 address=10.5.3.9 protocol=HTTP/1.1 server=- secure=true maxThreads=600 connectionTimeout=2 / -- !-- Define a SSL HTTP/1.1 Connector on port 8443 This connector uses the JSSE configuration, when using APR, the connector should be using the OpenSSL style configuration described in the APR documentation -- !-- Connector port=8443 protocol=HTTP/1.1 SSLEnabled=true maxThreads=150 scheme=https secure=true clientAuth=false sslProtocol=TLS / -- So It looks like the maxThreads has already been increased to 600. The server has been rebooted and tomcat stopped/started multiple times. Again all servers are Red Hat 5.2. Is there an additional parameter that needs to be set in order for tomcat to actually increase the maxThreads ? Thank You Keven Jones
Re: Tomcat maxThreads Issue
Caldarale, Charles R chuck.caldar...@unisys.com wrote in message news:0aae5ab84b013e45a7b61cb66943c17228fc438...@usea-exch7.na.uis.unisys.com... From: Jones, Keven [mailto:keven.jo...@ncr.com] Subject: RE: Tomcat maxThreads Issue Connector port=8011 enableLookups=false protocol=AJP/1.3 redirectPort=8443 maxThreads=400 connectionTimeout=2/ The syntax is ok, but I'd have to look at the code to see if connectionTimeout applies to anything other than the first request coming through. (AJP connections are intended to be persistent.) No, connectionTimeout applies to all requests. Any time that the Connector thread doesn't hear from httpd within 20 seconds (with this configuration) it will hang up and return to the pool. Note: this value also applies to reads from the request body. But I think almost all connections can send the default 8Kb in 20 seconds. The value for connectionTimeout looks low except for high-traffic sites (would go for something more like 5 minutes for a moderate traffic site). That would allow traffic spikes to die out without forcing unnecessary reconnections. You may also want to configure cping/cpong on the Apache side to aid mod_jk to detect that the connection has timed out. - Chuck - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
