Re: Tomcat maxThreads Issue
On 12.06.2009 00:13, Jones, Keven wrote: > This is a thread dump during a problem. I will take your advice and > execute 3 whan the next event happens. Thank you for your help! Then your problem doesn't last long enough. The presented dump doesn't contain stacks for 200 connector threads of one connector, so the problem (exhausted connector thread pool) was already over. Regards, Rainer > -Original Message- From: Rainer Jung > Sent: Thursday, June 11, 2009 6:08 PM To: > Tomcat Users List Subject: Re: Tomcat > maxThreads Issue > > > On 10.06.2009 18:26, Jones, Keven wrote: >> FYI, I'm just the linux admin as the "applicatons" group/developer >> and coder is not making any headway into resolving this. This is a >> mobility application (ie. Mobile banking, so you go to your iphone >> and go to your bank.mobi..)not that it matter but >> >> Ok, So yesterday I did perform a kill -3 and got a THREAD DUMP in >> the catalina.out file. I have copied most of it below: > > The dump during times where everything is fine doesn't reveal much. > You (unfortunately) need to do this, when the problem occurs. If you > manage to catch such an event, do 3 thread dumps, each a few seconds > apart from the next. That way one can also decide, whether a > situation is static or only short term. > > Regards, > > Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat maxThreads Issue
This is a thread dump during a problem. I will take your advice and execute 3 whan the next event happens. Thank you for your help! -Original Message- From: Rainer Jung Sent: Thursday, June 11, 2009 6:08 PM To: Tomcat Users List Subject: Re: Tomcat maxThreads Issue On 10.06.2009 18:26, Jones, Keven wrote: > FYI, I'm just the linux admin as the "applicatons" group/developer > and coder is not making any headway into resolving this. This is a > mobility application (ie. Mobile banking, so you go to your iphone > and go to your bank.mobi..)not that it matter but > > Ok, So yesterday I did perform a kill -3 and got a THREAD DUMP in the > catalina.out file. I have copied most of it below: The dump during times where everything is fine doesn't reveal much. You (unfortunately) need to do this, when the problem occurs. If you manage to catch such an event, do 3 thread dumps, each a few seconds apart from the next. That way one can also decide, whether a situation is static or only short term. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat maxThreads Issue
On 10.06.2009 18:26, Jones, Keven wrote: > FYI, I'm just the linux admin as the "applicatons" group/developer > and coder is not making any headway into resolving this. This is a > mobility application (ie. Mobile banking, so you go to your iphone > and go to your bank.mobi..)not that it matter but > > Ok, So yesterday I did perform a kill -3 and got a THREAD DUMP in the > catalina.out file. I have copied most of it below: The dump during times where everything is fine doesn't reveal much. You (unfortunately) need to do this, when the problem occurs. If you manage to catch such an event, do 3 thread dumps, each a few seconds apart from the next. That way one can also decide, whether a situation is static or only short term. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat maxThreads Issue
On 10.06.2009 21:25, Caldarale, Charles R wrote: >> From: Jones, Keven [mailto:keven.jo...@ncr.com] Subject: RE: Tomcat >> maxThreads Issue >> >> > redirectPort="8443" maxThreads="400" connectionTimeout="2"/> > > The syntax is ok, but I'd have to look at the code to see if > connectionTimeout applies to anything other than the first request > coming through. (AJP connections are intended to be persistent.) Haven't read through all of the thread yet, but yes, connectionTimeout on the AJP connector also closes the connection if the time between two requests on an existing connections is longer than connectionTimeout. 2 (20 seconds) is relatively short though. Setting it to something like 30 should be short enough. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat maxThreads Issue
> From: Jones, Keven [mailto:keven.jo...@ncr.com] > Subject: RE: Tomcat maxThreads Issue > > He is stating that If I put an address (the same address > Of the linux server that the tomcat instance is running on) > that that message will go away and the maxThreads will > actually move to 600. > > Anyone agree? Nope - your vendor is sadly mistaken or confused (or you misinterpreted what he said). The address (not server) attribute in a will limit the listen to just that IP address, whereas not including an address attribute results in listening on all IP addresses configured for the TCP/IP stack, including 10.10.10.10. The maxThreads attribute applies either way. Your vendor is also confused, because that's not the typically used between Tomcat and httpd; it's normally the AJP one, if you're using mod_jk or mod_proxy_ajp. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat maxThreads Issue
Update, I finally got my vendor on the line. He is telling me that this entry in my server.xml: Is the reason I'm seeing this in my catalina.out. He is stating that If I put an address (the same address Of the linux server that the tomcat instance is running on) that that message will go away and the maxThreads will actually move to 600. So it should look like where 10.10.10.10 is the internal IP of the tomcat app server Anyone agree? -Original Message- From: Caldarale, Charles R [mailto:chuck.caldar...@unisys.com] Sent: Wednesday, June 10, 2009 3:25 PM To: Tomcat Users List Subject: RE: Tomcat maxThreads Issue > From: Jones, Keven [mailto:keven.jo...@ncr.com] > Subject: RE: Tomcat maxThreads Issue > > enableLookups="false" protocol="AJP/1.3" redirectPort="8443" > maxThreads="400" connectionTimeout="2"/> The syntax is ok, but I'd have to look at the code to see if connectionTimeout applies to anything other than the first request coming through. (AJP connections are intended to be persistent.) - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat maxThreads Issue
"Caldarale, Charles R" wrote in message news:0aae5ab84b013e45a7b61cb66943c17228fc438...@usea-exch7.na.uis.unisys.com... >> From: Jones, Keven [mailto:keven.jo...@ncr.com] >> Subject: RE: Tomcat maxThreads Issue >> >> > enableLookups="false" protocol="AJP/1.3" redirectPort="8443" >> maxThreads="400" connectionTimeout="2"/> > >The syntax is ok, but I'd have to look at the code to see if >connectionTimeout applies to anything other than the first >request coming >through. (AJP connections are intended to be persistent.) > No, connectionTimeout applies to all requests. Any time that the Connector thread doesn't hear from httpd within 20 seconds (with this configuration) it will hang up and return to the pool. Note: this value also applies to reads from the request body. But I think almost all connections can send the default 8Kb in 20 seconds. The value for connectionTimeout looks low except for high-traffic sites (would go for something more like 5 minutes for a moderate traffic site). That would allow traffic spikes to die out without forcing unnecessary reconnections. You may also want to configure cping/cpong on the Apache side to aid mod_jk to detect that the connection has timed out. > - Chuck - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat maxThreads Issue
What does your Apache access_log say at the time when tomcat threads are exhausted ? I had a very similar issue in my environment. The culprit turned out to be JkMount options. Apache was sending all requests, whether it is for static or dynamic content to tomcat. At the same time my tomcats locked up, I was seeing a lot of requests for static content(gifs in my case) in the apache access_log Setting up mod_jk to only send *.jsp to tomcats fixed the issue for me. From: "Jones, Keven" To: "users@tomcat.apache.org" Sent: Wednesday, June 10, 2009 11:11:11 AM Subject: Tomcat maxThreads Issue Hello, I have Tomcat (can't figure out what version as the version.sh script is giving all kinds of errors) running on separate Linux servers. So an overview of my config is 1 web server, 4 app servers (tomcat), and 1 db server. We were seeing an increase in the httpd processes on the web server to the point where I had to increase the MaxClients on the apache 2 server to 2048. I finally figured out that the normal number of httpd daemons on the web server (50-100) was spiking due to tomcat processes stalling or going completely down. So I started looking at why tomcat was failing and I found this error message in the ACStarError.log : SEVERE: All threads (200) are currently busy, waiting. Increase maxThreads (200) or check the servlet status. So I checked the server.xml file and found the following: So It looks like the maxThreads has already been increased to 600. The server has been rebooted and tomcat stopped/started multiple times. Again all servers are Red Hat 5.2. Is there an additional parameter that needs to be set in order for tomcat to actually increase the maxThreads ? Thank You Keven Jones
RE: Tomcat maxThreads Issue
> From: Jones, Keven [mailto:keven.jo...@ncr.com] > Subject: RE: Tomcat maxThreads Issue > > enableLookups="false" protocol="AJP/1.3" redirectPort="8443" > maxThreads="400" connectionTimeout="2"/> The syntax is ok, but I'd have to look at the code to see if connectionTimeout applies to anything other than the first request coming through. (AJP connections are intended to be persistent.) - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat maxThreads Issue
Does the syntax of this look correct to everyone? I have added maxThreads="400" and connectionTimeout="2"(not sure if I need the connectionTimeout) Thanks everyone -Original Message- From: Tim Funk [mailto:funk...@apache.org] Sent: Wednesday, June 10, 2009 1:02 PM To: Tomcat Users List Subject: Re: Tomcat maxThreads Issue Looking at the history of the thread - here are some guesses You have a connector on port 81 (http) and a connector on 8011 (ajp) and commented out connector for 443. The port 81 connector and commented out connector are set to 600 - which means absolutely nothing if apache is talking to tomcat via jk. Set |maxThreads on the connector for using port 8011 and life should be better. If there are mulitple apache servers in front of tomcat - you need to ensure you have (#apaches * max number connections per apache) threads available on each tomcat. The thread history seems to indicate you only have one apache with 100 workers max. So raising the maxThreads in tomcat from 200 to something higher is not needed. Is there a firewall between apache and tomcat? If so - it could mean that your firewall is timing out the connection and then bad things happen. | http://tomcat.apache.org/tomcat-6.0-doc/config/ajp.html -Tim - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat maxThreads Issue
> From: Caldarale, Charles R > Subject: RE: Tomcat maxThreads Issue > > There has been much discussion about this of late on this mailing list; > I'll try to find some pointers for you. This is a useful reference: http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html#Firewall%20Connection%20Dropping The best (and unfortunately, biggest) discussion of this issue is here: http://marc.info/?t=12422502151&r=1&w=2 The most useful messages in this 61-posting thread are Rainer's, should you want to browse through them. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat maxThreads Issue
> From: Jones, Keven [mailto:keven.jo...@ncr.com] > Subject: RE: Tomcat maxThreads Issue > I see we have Engine set to Standalone, > And I see the Service is catalina (?) I would leave the name as Catalina (it's case sensitive), since that's what everyone expects (unless you have multiple elements - a very rare need). - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat maxThreads Issue
> From: Jonathan Mast [mailto:jhmast.develo...@gmail.com] > Subject: Re: Tomcat maxThreads Issue > > Add the following somewhere near the top (I place mine after all the > explanatory comments): > JAVA_OPTS=" -Xmx1024m" Again, if there's no evidence of a heap issue (and none has been presented) *do not* change the heap settings - at best that can only muddy the water. If you want to look at garbage collection activity, use -verbose:gc (which should work even on JRockit) to get information about how often a GC occurs. Unless it's happening very frequently, leave the heap settings alone. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat maxThreads Issue
Cool, Also, I'm examining my tomcat server.xml conf file for each instnace on each server. I am finding that Tomcat1 AJP connector is 8011, Tomcat2 AJP 8012, Tomcat3 AJP 8013, and Tomcat4 AJP 8014. So I will try and Add a maxThreads 600 on each of these. Thanks -Original Message- From: Tim Funk [mailto:funk...@apache.org] Sent: Wednesday, June 10, 2009 1:33 PM To: Tomcat Users List Subject: Re: Tomcat maxThreads Issue If your not scared of the performance penalty (which these days isn't scary unless you really have high load) - try adding this to your apache config - it will end the connection between apache and tomcat after each request. This virtually eliminates the chance of a firewall dropping connections for you. JkOptions +DisableReuse YOu prob dont want this permanantly but its a good test http://tomcat.apache.org/connectors-doc/reference/apache.html -Tim Jones, Keven wrote: > Thank you Tim > > I am going to attempt to set the maxThreads on 8011 to 600 and hope it helps. > > The config is 1 linux apache2 server with 4 linux tomcat servers and each > tomcat server has 4 sepearte tomcat instances. So we have a total of 16 > tomcat instances across 4 servers. > > Yes, there is a firewall between the web sever and the app(tomcat)/db(mysql) > server. > > KJ > > -Original Message- > From: Tim Funk [mailto:funk...@apache.org] > Sent: Wednesday, June 10, 2009 1:02 PM > To: Tomcat Users List > Subject: Re: Tomcat maxThreads Issue > > Looking at the history of the thread - here are some guesses > > You have a connector on port 81 (http) and a connector on 8011 (ajp) and > commented out connector for 443. > > The port 81 connector and commented out connector are set to 600 - which > means absolutely nothing if apache is talking to tomcat via jk. > > Set |maxThreads on the connector for using port 8011 and life should be > better. > > If there are mulitple apache servers in front of tomcat - you need to ensure > you have (#apaches * max number connections per apache) threads available on > each tomcat. The thread history seems to indicate you only have one apache > with 100 workers max. So raising the maxThreads in tomcat from 200 to > something higher is not needed. > > Is there a firewall between apache and tomcat? If so - it could mean that > your firewall is timing out the connection and then bad things happen. > | > http://tomcat.apache.org/tomcat-6.0-doc/config/ajp.html > > > -Tim > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > > > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat maxThreads Issue
Well I'm not sure how to check the current memory settings but I sure know how to adjust them :) Presuming that you downloaded Tomcat directly and installed manually: open "$TOMCAT/bin/catalina.sh" Add the following somewhere near the top (I place mine after all the explanatory comments): JAVA_OPTS=" -Xmx1024m" This is tells java to allow a maximum of 1024 megabytes of memory to this instance of tomcat. Adjust appropriately. Please note: * the leading SPACE before the "-Xmx". * this is not a labeled parameter, therefore no "=" btwn the option and value. On Wed, Jun 10, 2009 at 1:12 PM, Jones, Keven wrote: > Thank you Tim > > I am going to attempt to set the maxThreads on 8011 to 600 and hope it > helps. > > The config is 1 linux apache2 server with 4 linux tomcat servers and each > tomcat server has 4 sepearte tomcat instances. So we have a total of 16 > tomcat instances across 4 servers. > > Yes, there is a firewall between the web sever and the > app(tomcat)/db(mysql) server. > > KJ > > -Original Message- > From: Tim Funk [mailto:funk...@apache.org] > Sent: Wednesday, June 10, 2009 1:02 PM > To: Tomcat Users List > Subject: Re: Tomcat maxThreads Issue > > Looking at the history of the thread - here are some guesses > > You have a connector on port 81 (http) and a connector on 8011 (ajp) and > commented out connector for 443. > > The port 81 connector and commented out connector are set to 600 - which > means absolutely nothing if apache is talking to tomcat via jk. > > Set |maxThreads on the connector for using port 8011 and life should be > better. > > If there are mulitple apache servers in front of tomcat - you need to > ensure you have (#apaches * max number connections per apache) threads > available on each tomcat. The thread history seems to indicate you only have > one apache with 100 workers max. So raising the maxThreads in tomcat from > 200 to something higher is not needed. > > Is there a firewall between apache and tomcat? If so - it could mean that > your firewall is timing out the connection and then bad things happen. > | > http://tomcat.apache.org/tomcat-6.0-doc/config/ajp.html > > > -Tim > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
RE: Tomcat maxThreads Issue
> From: Jones, Keven [mailto:keven.jo...@ncr.com] > Subject: RE: Tomcat maxThreads Issue > > I am going to attempt to set the maxThreads on 8011 > to 600 and hope it helps. It will likely only postpone the problem. > Yes, there is a firewall between the web sever and the > app(tomcat)/db(mysql) server. The question was whether or not there's a firewall between httpd and Tomcat, not Tomcat and the DB server. If there is a firewall between httpd and Tomcat, it could be silently dropping the AJP connections, leaving Tomcat threads stranded on unusable ports. There has been much discussion about this of late on this mailing list; I'll try to find some pointers for you. Unless you're seeing OutOfMemoryErrors in your Tomcat logs, the heap sizing is very likely a red herring. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat maxThreads Issue
If your not scared of the performance penalty (which these days isn't scary unless you really have high load) - try adding this to your apache config - it will end the connection between apache and tomcat after each request. This virtually eliminates the chance of a firewall dropping connections for you. JkOptions +DisableReuse YOu prob dont want this permanantly but its a good test http://tomcat.apache.org/connectors-doc/reference/apache.html -Tim Jones, Keven wrote: Thank you Tim I am going to attempt to set the maxThreads on 8011 to 600 and hope it helps. The config is 1 linux apache2 server with 4 linux tomcat servers and each tomcat server has 4 sepearte tomcat instances. So we have a total of 16 tomcat instances across 4 servers. Yes, there is a firewall between the web sever and the app(tomcat)/db(mysql) server. KJ -Original Message- From: Tim Funk [mailto:funk...@apache.org] Sent: Wednesday, June 10, 2009 1:02 PM To: Tomcat Users List Subject: Re: Tomcat maxThreads Issue Looking at the history of the thread - here are some guesses You have a connector on port 81 (http) and a connector on 8011 (ajp) and commented out connector for 443. The port 81 connector and commented out connector are set to 600 - which means absolutely nothing if apache is talking to tomcat via jk. Set |maxThreads on the connector for using port 8011 and life should be better. If there are mulitple apache servers in front of tomcat - you need to ensure you have (#apaches * max number connections per apache) threads available on each tomcat. The thread history seems to indicate you only have one apache with 100 workers max. So raising the maxThreads in tomcat from 200 to something higher is not needed. Is there a firewall between apache and tomcat? If so - it could mean that your firewall is timing out the connection and then bad things happen. | http://tomcat.apache.org/tomcat-6.0-doc/config/ajp.html -Tim - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat maxThreads Issue
Thank you Tim I am going to attempt to set the maxThreads on 8011 to 600 and hope it helps. The config is 1 linux apache2 server with 4 linux tomcat servers and each tomcat server has 4 sepearte tomcat instances. So we have a total of 16 tomcat instances across 4 servers. Yes, there is a firewall between the web sever and the app(tomcat)/db(mysql) server. KJ -Original Message- From: Tim Funk [mailto:funk...@apache.org] Sent: Wednesday, June 10, 2009 1:02 PM To: Tomcat Users List Subject: Re: Tomcat maxThreads Issue Looking at the history of the thread - here are some guesses You have a connector on port 81 (http) and a connector on 8011 (ajp) and commented out connector for 443. The port 81 connector and commented out connector are set to 600 - which means absolutely nothing if apache is talking to tomcat via jk. Set |maxThreads on the connector for using port 8011 and life should be better. If there are mulitple apache servers in front of tomcat - you need to ensure you have (#apaches * max number connections per apache) threads available on each tomcat. The thread history seems to indicate you only have one apache with 100 workers max. So raising the maxThreads in tomcat from 200 to something higher is not needed. Is there a firewall between apache and tomcat? If so - it could mean that your firewall is timing out the connection and then bad things happen. | http://tomcat.apache.org/tomcat-6.0-doc/config/ajp.html -Tim - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat maxThreads Issue
I did a grep for heap in catalina.out but do not see any out of memory entries. How do I check the default JVM memory setting and how do I adjust it? Thx -Original Message- From: Jonathan Mast [mailto:jhmast.develo...@gmail.com] Sent: Wednesday, June 10, 2009 1:04 PM To: Tomcat Users List Subject: Re: Tomcat maxThreads Issue I encountered a similar issue of numerous Httpd processes spawning b/c of Tomcat. The issue for me was that Tomcat was running out of memory, and Httpd for some reason thought spawning a bunch a threads would help ;) Do you know how much memory these instances of Tomcat are using? The default JVM memory is ridiculously low and requires the adjustment to higher maximum value for production servers. Grep "heap" catalina.out to see if tomcat is indeed running out of memory. On Wed, Jun 10, 2009 at 12:44 PM, Peter Crowther < peter.crowt...@melandra.com> wrote: > OK, so that's a JRockit JVM, not a Sun JVM. Remind me not to point > you at any more Sun documentation! > > As a rough look at that, there are a few threads in the HTTP connector > thread pool, a bunch of threads reading data from JK connections, and > one or two in your app code. I've never looked at the innards of JK, > so will have to leave that to one of the JK experts to interpret. > >- Peter > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat maxThreads Issue
I encountered a similar issue of numerous Httpd processes spawning b/c of Tomcat. The issue for me was that Tomcat was running out of memory, and Httpd for some reason thought spawning a bunch a threads would help ;) Do you know how much memory these instances of Tomcat are using? The default JVM memory is ridiculously low and requires the adjustment to higher maximum value for production servers. Grep "heap" catalina.out to see if tomcat is indeed running out of memory. On Wed, Jun 10, 2009 at 12:44 PM, Peter Crowther < peter.crowt...@melandra.com> wrote: > OK, so that's a JRockit JVM, not a Sun JVM. Remind me not to point you at > any more Sun documentation! > > As a rough look at that, there are a few threads in the HTTP connector > thread pool, a bunch of threads reading data from JK connections, and one or > two in your app code. I've never looked at the innards of JK, so will have > to leave that to one of the JK experts to interpret. > >- Peter > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: Tomcat maxThreads Issue
Looking at the history of the thread - here are some guesses You have a connector on port 81 (http) and a connector on 8011 (ajp) and commented out connector for 443. The port 81 connector and commented out connector are set to 600 - which means absolutely nothing if apache is talking to tomcat via jk. Set |maxThreads on the connector for using port 8011 and life should be better. If there are mulitple apache servers in front of tomcat - you need to ensure you have (#apaches * max number connections per apache) threads available on each tomcat. The thread history seems to indicate you only have one apache with 100 workers max. So raising the maxThreads in tomcat from 200 to something higher is not needed. Is there a firewall between apache and tomcat? If so - it could mean that your firewall is timing out the connection and then bad things happen. | http://tomcat.apache.org/tomcat-6.0-doc/config/ajp.html -Tim - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat maxThreads Issue
OK, so that's a JRockit JVM, not a Sun JVM. Remind me not to point you at any more Sun documentation! As a rough look at that, there are a few threads in the HTTP connector thread pool, a bunch of threads reading data from JK connections, and one or two in your app code. I've never looked at the innards of JK, so will have to leave that to one of the JK experts to interpret. - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat maxThreads Issue
ng/Thread.run(Thread.java:595)[optimized] at jrockit/vm/RNI.c2java()V(Native Method) -- end of trace "aRequest: -Completed-" id=49500 idx=0x18e tid=22045 prio=5 alive, in native, waiting, daemon -- Waiting for notification on: org/apache/tomcat/util/threads/threadpool$controlrunna...@0x198 12ae0[fat lock] at jrockit/vm/Threads.waitForSignal(J)Z(Native Method) at jrockit/vm/Locks.wait(Ljava/lang/Object;J)V(Unknown Source)[inlined] at java/lang/Object.wait()V(Native Method)[optimized] at org/apache/tomcat/util/threads/ThreadPool$ControlRunnable.run(ThreadPool.java:658) ^-- Lock released while waiting: org/apache/tomcat/util/threads/threadpool$controlrunna...@0x19 812ae0[fat lock] at java/lang/Thread.run(Thread.java:595)[optimized] at jrockit/vm/RNI.c2java()V(Native Method) -- end of trace "aRequest: -Completed-" id=49508 idx=0x196 tid=22091 prio=5 alive, in native, waiting, daemon -- Waiting for notification on: org/apache/tomcat/util/threads/threadpool$controlrunna...@0x198 73a80[fat lock] at jrockit/vm/Threads.waitForSignal(J)Z(Native Method) at jrockit/vm/Locks.wait(Ljava/lang/Object;J)V(Unknown Source)[inlined] at java/lang/Object.wait()V(Native Method)[optimized] at org/apache/tomcat/util/threads/ThreadPool$ControlRunnable.run(ThreadPool.java:658) ^-- Lock released while waiting: org/apache/tomcat/util/threads/threadpool$controlrunna...@0x19 873a80[fat lock] at java/lang/Thread.run(Thread.java:595)[optimized] at jrockit/vm/RNI.c2java()V(Native Method) -- end of trace = END OF THREAD DUMP === -Original Message- From: Peter Crowther [mailto:peter.crowt...@melandra.com] Sent: Wednesday, June 10, 2009 12:16 PM To: 'Tomcat Users List' Subject: RE: Tomcat maxThreads Issue > From: Jones, Keven [mailto:keven.jo...@ncr.com] How do I > > A. get thread dumps when the processes are dying The top of http://java.sun.com/developer/technicalArticles/Programming/Stacktrace/ is good general reading. kill -QUIT If ps is confused by all the threads, depending on your Java version, you might be able to use jps (http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jps.html) to get the pid. Depending on your Java version, jstack (http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jstack.html) will also give you thread dumps. - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat maxThreads Issue
> From: Jones, Keven [mailto:keven.jo...@ncr.com] > How do I > > A. get thread dumps when the processes are dying The top of http://java.sun.com/developer/technicalArticles/Programming/Stacktrace/ is good general reading. kill -QUIT If ps is confused by all the threads, depending on your Java version, you might be able to use jps (http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jps.html) to get the pid. Depending on your Java version, jstack (http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jstack.html) will also give you thread dumps. - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat maxThreads Issue
I don't have any tomcat knowledge. Thank you for the reply. How do I A. get thread dumps when the processes are dying And B. You mentioned the engine and service name. I see we have Engine set to Standalone, And I see the Service is catalina (?) which should I adjust? Thx again -Original Message- From: Caldarale, Charles R [mailto:chuck.caldar...@unisys.com] Sent: Wednesday, June 10, 2009 11:33 AM To: Tomcat Users List Subject: RE: Tomcat maxThreads Issue > From: Jones, Keven [mailto:keven.jo...@ncr.com] > Subject: RE: Tomcat maxThreads Issue > > > enableLookups="false" protocol="AJP/1.3" redirectPort="8443" > /> This is the connector Mark was referring to (he should have said AJP, not APR). Since you have no maxThreads attribute, it defaults to 200, which may or may not be appropriate for your expected load. The real question is what are those 200 threads doing? Try taking a couple of thread dumps when you get into the problem and see what they're stuck on. > jvmRoute="mweb3a"> You really should have the and names the same, to keep JMX happy. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat maxThreads Issue
> From: Jones, Keven [mailto:keven.jo...@ncr.com] > Subject: RE: Tomcat maxThreads Issue > > > enableLookups="false" protocol="AJP/1.3" redirectPort="8443" /> This is the connector Mark was referring to (he should have said AJP, not APR). Since you have no maxThreads attribute, it defaults to 200, which may or may not be appropriate for your expected load. The real question is what are those 200 threads doing? Try taking a couple of thread dumps when you get into the problem and see what they're stuck on. > jvmRoute="mweb3a"> You really should have the and names the same, to keep JMX happy. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat maxThreads Issue
Sorry, here is the complete server.xml file: -Original Message- From: Peter Crowther [mailto:peter.crowt...@melandra.com] Sent: Wednesday, June 10, 2009 11:19 AM To: 'Tomcat Users List' Subject: RE: Tomcat maxThreads Issue > From: Jones, Keven [mailto:keven.jo...@ncr.com] Here is the complete > config file. Thanks for the quick reply. The list strips attachments - please paste inline. - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat maxThreads Issue
> From: Jones, Keven [mailto:keven.jo...@ncr.com] > Here is the complete config file. Thanks for the quick reply. The list strips attachments - please paste inline. - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat maxThreads Issue
Here is the complete config file. Thanks for the quick reply. -Original Message- From: Mark Thomas [mailto:ma...@apache.org] Sent: Wednesday, June 10, 2009 11:14 AM To: Tomcat Users List Subject: Re: Tomcat maxThreads Issue Jones, Keven wrote: > Hello, > > I have Tomcat (can't figure out what version as the version.sh script > is giving all kinds of errors) running on separate Linux servers. So > an overview of my config is > 1 web server, 4 app servers (tomcat), and 1 db server. > > We were seeing an increase in the httpd processes on the web server to the > point where I had to increase the MaxClients on the apache 2 server to 2048. > I finally figured out that the normal number of httpd daemons on the web > server (50-100) was spiking due to tomcat processes stalling or going > completely down. > > So I started looking at why tomcat was failing and I found this error message > in the ACStarError.log : > > > SEVERE: All threads (200) are currently busy, waiting. Increase maxThreads > (200) or check the servlet status. > So I checked the server.xml file and found the following: > >server="-" > maxThreads="600" connectionTimeout="2" >redirectPort="443" /> > > > > > > So It looks like the maxThreads has already been increased to 600. The server > has been rebooted and tomcat stopped/started multiple times. Again all > servers are Red Hat 5.2. Is there an additional parameter that needs to be > set in order for tomcat to actually increase the maxThreads ? > Thank You That is the http connector. What about the APR connector? You can leave out stuff that is commented out. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat maxThreads Issue
Jones, Keven wrote: > Hello, > > I have Tomcat (can't figure out what version as the version.sh script is > giving all kinds of errors) running on separate Linux servers. So an overview > of my config is > 1 web server, 4 app servers (tomcat), and 1 db server. > > We were seeing an increase in the httpd processes on the web server to the > point where I had to increase the MaxClients on the apache 2 server to 2048. > I finally figured out that the normal number of httpd daemons on the web > server (50-100) was spiking due to tomcat processes stalling or going > completely down. > > So I started looking at why tomcat was failing and I found this error message > in the ACStarError.log : > > > SEVERE: All threads (200) are currently busy, waiting. Increase maxThreads > (200) or check the servlet status. > So I checked the server.xml file and found the following: > >server="-" > maxThreads="600" connectionTimeout="2" >redirectPort="443" /> > > > > > > So It looks like the maxThreads has already been increased to 600. The server > has been rebooted and tomcat stopped/started multiple times. Again all > servers are Red Hat 5.2. Is there an additional parameter that needs to be > set in order for tomcat to actually increase the maxThreads ? > Thank You That is the http connector. What about the APR connector? You can leave out stuff that is commented out. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Tomcat maxThreads Issue
Hello, I have Tomcat (can't figure out what version as the version.sh script is giving all kinds of errors) running on separate Linux servers. So an overview of my config is 1 web server, 4 app servers (tomcat), and 1 db server. We were seeing an increase in the httpd processes on the web server to the point where I had to increase the MaxClients on the apache 2 server to 2048. I finally figured out that the normal number of httpd daemons on the web server (50-100) was spiking due to tomcat processes stalling or going completely down. So I started looking at why tomcat was failing and I found this error message in the ACStarError.log : SEVERE: All threads (200) are currently busy, waiting. Increase maxThreads (200) or check the servlet status. So I checked the server.xml file and found the following: So It looks like the maxThreads has already been increased to 600. The server has been rebooted and tomcat stopped/started multiple times. Again all servers are Red Hat 5.2. Is there an additional parameter that needs to be set in order for tomcat to actually increase the maxThreads ? Thank You Keven Jones
