After some digging thru config files setup by the vendor, I think I might have found the root cause ..Correlating the access_log and tomcat logs, I found out that tomcat threads were shooting up whenever a large number of GIF files were being requested.
These are the modjk settings in our application config files that are included in httpd.conf JkMount /LCSW/* ajp13 JkMount /LCSW/*.jsp ajp13 JkMount /LCSW/*.jsp/* ajp13 So when an image is accessed via the following URL, http://blahbhal/LCSW/images/header.gif , the above routes were sending it to tomcat. I verified this by shutting down tomcat and accessing the above URL, and was unsuccessful. Now I plan to add the following excludes JkUnMount /*.htm ajp13 JkUnMount /*.html ajp13 JkUnMount /*.png ajp13 JkUnMount /*.gif ajp13 JkUnMount /*.jpg ajp13 I tested the above in a test environment and was able to access a gif file with tomcat down. my question is, how does the ajp13 connector interpret a gif file/static content request as opposed to a jsp request? Chetan ________________________________ From: Chetan Chheda <chetan_chh...@yahoo.com> To: Tomcat Users List <users@tomcat.apache.org> Sent: Monday, May 18, 2009 5:53:07 PM Subject: Re: Tomcat not closing threads Hi martin, Thanks for your reply but you have left me confused Since Apache is setup as below Maxclients = 512 ThreadsPerChild = 256 Wont it restrict the processes to 2 processes with 256 threads each instead of the ServerLimit(16) ? Looking at the OS , I see 2 httpd processes each with 258 threads Thanks, Chetan ________________________________ From: Martin Gainty <mgai...@hotmail.com> To: Tomcat Users List <users@tomcat.apache.org> Sent: Monday, May 18, 2009 4:00:16 PM Subject: RE: Tomcat not closing threads good to know With worker mpm the connection_pool_size defaults to ThreadsPerChild ThreadsPerChild that in the final gives the MaxClients connections to the Tomcat. So the default value for connection_pool_size is always ThreadsPerChild For threaded and hybrid servers (e.g. beos or worker) MaxClients restricts the total number of threads that will be available to serve clients. The default value for beos is 50. For hybrid MPMs the default value is 16 (ServerLimit) multiplied by the value of 25 (ThreadsPerChild). Therefore, to increase MaxClients to a value that requires more than 16 processes, you must also raise ServerLimit. /*******in your case*******/ (MaxClients)MaxClients=16(ServerLimit)*256(ThreadsPerChild)=4096 onto thread calculation: Description:Minimum number of idle threads available to handle request spikes Syntax:MinSpareThreads number Default:See usage for details Context:server config Status:MPM Module:beos, mpm_netware, mpmt_os2, worker Minimum number of idle threads to handle request spikes. Different MPMs deal with this directive differently. worker uses a default of MinSpareThreads 75 and deal with idle threads on a server-wide basis. If there aren't enough idle threads in the server then child processes are created until the number of idle threads is greater than number. mpm_netware uses a default of MinSpareThreads 10 and, since it is a single-process MPM, tracks this on a server-wide bases. beos and mpmt_os2 work similar to mpm_netware. The default for beos is MinSpareThreads 1. For mpmt_os2 the default value is 5. /*********lets assume you havent touched this and MinSpareThreads stays at 75***********/ MaxSpareThreads(this is the parameter that is plaguing your environment) The range of the MaxSpareThreads value is restricted. Apache will correct the given value automatically according to the following rules: perchild requires MaxSpareThreads to be less or equal than ThreadLimit.mpm_netware wants the value to be greater than MinSpareThreads.For leader, threadpool and worker the value must be greater or equal than the sum of MinSpareThreads and ThreadsPerChild. MaxSpareThreads >= MinSpareThreads+ThreadsPerChild MaxSpareThreads >=75+256 MaxSpareThreads must be >= 331 after all that if you still dont have enough SpareThreads i would up (master thread limit) ThreadLimit to the max The default value for ThreadLimit is 1920 when used with mpm_winnt and 64 when used with the others. There is a hard limit of ThreadLimit 20000 (or ThreadLimit 15000 with mpm_winnt) compiled into the server. This is intended to avoid nasty effects caused by typos. /****conceivable to start with 1920 and increase by increments of 500 until number of threads sufficient ***/ BTW: the ceiling on ThreadLimit=15000 In your case ThreadLimit=1920 HTH Martin Gainty ______________________________________________ Jogi és Bizalmassági kinyilatkoztatás/Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité Ez az üzenet bizalmas. Ha nem ön az akinek szánva volt, akkor kérjük, hogy jelentse azt nekünk vissza. Semmiféle továbbítása vagy másolatának készítése nem megengedett. Ez az üzenet csak ismeret cserét szolgál és semmiféle jogi alkalmazhatósága sincs. Mivel az electronikus üzenetek könnyen megváltoztathatóak, ezért minket semmi felelöség nem terhelhet ezen üzenet tartalma miatt. Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen. Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité pour le contenu fourni. > Date: Mon, 18 May 2009 12:03:35 -0700 > From: chetan_chh...@yahoo.com > Subject: Re: Tomcat not closing threads > To: users@tomcat.apache.org > > Apache is using worker mpm > Server version: Apache/2.0.59 HP-UX_Apache-based_Web_Server > Server built: Aug 21 2007 13:59:06 > Server's Module Magic Number: 20020903:12 > Server loaded: APR 0.9.12, APR-UTIL 0.9.12 > Compiled using: APR 0.9.12, APR-UTIL 0.9.12 > Architecture: 64-bit > Server compiled with.... > -D APACHE_MPM_DIR="server/mpm/worker" > > > > > ________________________________ > From: Martin Gainty <mgai...@hotmail.com> > To: Tomcat Users List <users@tomcat.apache.org> > Sent: Monday, May 18, 2009 2:44:45 PM > Subject: RE: Tomcat not closing threads > > > need to know if Apache MPM prefork or Apache worker is configured > $APACHE_HOME/bin/Apache -l > > *any compiled-in modules with 'mpm' in the module-name is prefork* > *any compiled-in modules with 'worker' in the module-name is worker* > > Martin Gainty > ______________________________________________ > Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité > > Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger > sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung > oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich > dem Austausch von Informationen und entfaltet keine rechtliche > Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen > wir keine Haftung fuer den Inhalt uebernehmen. > Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le > destinataire prévu, nous te demandons avec bonté que pour satisfaire informez > l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci > est interdite. Ce message sert à l'information seulement et n'aura pas > n'importe quel effet légalement obligatoire. Étant donné que les email > peuvent facilement être sujets à la manipulation, nous ne pouvons accepter > aucune responsabilité pour le contenu fourni. > > > > > > Date: Mon, 18 May 2009 08:45:43 -0700 > > From: chetan_chh...@yahoo.com > > Subject: Re: Tomcat not closing threads > > To: users@tomcat.apache.org > > > > So what you are suggesting is my mod_jk and tomcat parameters in server.xml > > are messed up .. > > > > Here is my current setup I have one apache load balancing to 2 tomcat > > workers > > Maxclients = 512 > > ThreadsPerChild = 256 > > Connection_pool_size = 256 > > Tomcat MaxThreads = 150 > > > > I will suggest my team to change this to ... > > connection_pool_size = 120 - which means that I will have 120*2(2 apache > > server procs) = 240 threads per tomcat > > Tomcat maxthreads = 300 > > > > So out of total 512 worker threads , a max of 480 threads will be used by > > mod_jk leaving the rest for serving static content. > > > > is my math correct? > > > > Thanks! > > > > > > > > ________________________________ > > From: Martin Gainty <mgai...@hotmail.com> > > To: Tomcat Users List <users@tomcat.apache.org> > > Sent: Monday, May 18, 2009 11:23:58 AM > > Subject: RE: Tomcat not closing threads > > > > > > some documentation: > > > > http://httpd.apache.org/docs/2.0/mod/mpm_common.html > > MaxClients: > > The MaxClients directive sets the limit on the number of simultaneous > > requests that will be served. > > Any connection attempts over the MaxClients limit will normally be queued, > > up to a number based on the ListenBacklog directive. > > Once a child process is freed at the end of a different request, the > > connection will then be serviced. > > For non-threaded servers (i.e., prefork), MaxClients translates into the > > maximum number of child processes that will be launched to serve requests. > > The default value is 256; to increase it, you must also raise ServerLimit. > > For threaded and hybrid servers (e.g. beos or worker) MaxClients restricts > > the total number of threads that will be available to serve clients. > > The default value for beos is 50. > > For hybrid MPMs (your situation) the default value is 16 (ServerLimit) > > multiplied by the value of ThreadsPerChild. > > Therefore, to increase MaxClients to a value that requires more than 16 > > processes, you must also raise ServerLimit. > > > > ThreadsPerChild > > This directive sets the number of threads created by each child process. > > The child creates these threads at startup and never creates more. > > If using an MPM like mpm_winnt, where there is only one child process, this > > number should be high enough to handle the entire load of the server. > > If using an MPM like worker, where there are multiple child processes, the > > total number of threads should be high enough to handle the common load on > > the server. > > The default value for ThreadsPerChild is 64 when used with mpm_winnt > > The default value is 25 when used with other configurations such as worker > > > > also from mladens discussion of connection_pool_size calcs based on Apache > > HTTPD Prefork vs Worker > > http://www.mail-archive.com/users@tomcat.apache.org/msg14767.html > > Apache Prefork configuration > > Apache 1.3 and Apache 2.x-prefork create a separate child process for > > each client connection, meaning that if you have default > > 250 MaxClients, you can end up with MaxClients * connection_pool_size > > connections to the Tomcat, that would in your case be 250000. > > (and in the case of ThreadsPerChild parameter > > MaxClients*64*connection_pool_size) > > > > Apache Worker configuration > > With worker mpm the connection_pool_size defaults to ThreadsPerChild, > > that in the final gives the MaxClients connections to the Tomcat. > > So the default value for connection_pool_size is always ThreadsPerChild > > that with prefork mpm is by design always 1. > > (in the case of ThreadsPerChild parameter not being null > > MaxClients*ThreadsPerChild) > > > > HTH > > Martin > > ______________________________________________ > > Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité > > > > Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene > > Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte > > Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht > > dient lediglich dem Austausch von Informationen und entfaltet keine > > rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von > > E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen. > > Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le > > destinataire prévu, nous te demandons avec bonté que pour satisfaire > > informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie > > de ceci est interdite. Ce message sert à l'information seulement et n'aura > > pas n'importe quel effet légalement obligatoire. Étant donné que les email > > peuvent facilement être sujets à la manipulation, nous ne pouvons accepter > > aucune responsabilité pour le contenu fourni. > > > > > > > > > > Date: Mon, 18 May 2009 07:43:04 -0700 > > From: chetan_chh...@yahoo.com > > Subject: Re: Tomcat not closing threads > > To: users@tomcat.apache.org > > > > > > > > Our network team is already engaged and looking into this. We had several > > instances of tomcats locking up today. I have attached one such thread dump > > here .. > > > > Can some of you pls take a look and point me into the right direction? > > > > > > > > > > > > > > From: rad muthu <rmuth...@gmail.com> > > To: Tomcat Users List <users@tomcat.apache.org> > > Sent: Wednesday, May 13, 2009 8:05:08 PM > > Subject: Re: Tomcat not closing threads > > > > Chetan, > > > > > > We had similar issue with Sun solaris and weblogic. > > > > My system admin found out there is a socket read errors are going on . He > > used snoop command in solaris. After tuning the TCP parameters the issue got > > resolved. > > > > You might also need to something like that. > > > > Thanks > > Krish > > > > > > > > On Wed, May 13, 2009 at 11:57 AM, Chetan Chheda > > <chetan_chh...@yahoo.com>wrote: > > > > > I guess I need to ask my question again. Im primarily an > > HP-UX > > > administrator and recently inherited this web based application ... > > > > > > We are having some issues in accomodating additional user > > > load/functionality. While a part of the team is looking at optimizing > > > code, > > > I am responsible for infrastructure componants. > > > > > > One particular focus area for me is the mod_jk and tomcat configurations. > > > 1. Last of April saw one of the tomcat's stop processing requests. Looking > > > at the mod_jk.log I saw the following errors > > > Unable to get the free endpoint for worker XXX from 37 slots .... > > > I have setup the status servlet for mod_jk in which I saw that max slots > > > were opened. Thinking that mod_jk needs more number of connections, I > > > basically set it to ThreadsPerChild from worker MPM > > > > > > 2. Now we are randomly seeing that even a simple activity as user login > > > causes a spike in used tomcat connector threads. > > Eventually all tomcat > > > threads are used and no new requests can be serviced. > > > > > > So, long story short, is my apache, mod_jk and tomcat configuration in > > > sync? > > > Maxclients = 512 > > > ThreadsPerChild = 256 > > > So that means 2 server processes with 256 threads each. > > > > > > I have 2 worker threads in my worker.properties file with > > > connection_pool_size=256. > > > Does this mean that total number of connections into tomcat = 256 * > > > 2(number of workers) * 2(number of server procs) = 1024 ?? > > > > > > Do I need to change the connection_pool_size to 120 . Which means 120*2 *2 > > > = 480 and that leaves the remainder of apache threads for static content? > > > > > > 3. I have worker.XXX.cache_timeout=900 and no connectionTimeout in > > > server.xml . Is this why connections remain open? > > > > > > Thanks, > > > Chetan > > > > > > > > > > > ________________________________ > > > From: "Caldarale, Charles R" <chuck.caldar...@unisys.com> > > > To: Tomcat Users List <users@tomcat.apache.org> > > > Sent: Tuesday, May 12, 2009 10:47:00 PM > > > Subject: RE: Tomcat not closing threads > > > > > > > From: Chetan Chheda [mailto:chetan_chh...@yahoo.com] > > > > Subject: Re: Tomcat not closing threads > > > > > > > > I managed to get a thread dump during one such tomcat hangs. Most of > > > > the threads are in the following status ... > > > > > > The ones you show are simply waiting for input from httpd; that's a pretty > > > normal state. When looking at a thread dump, it's often > > the oddball thread > > > that's not doing what the rest are that is causing a problem. However, if > > > all of the threads are just waiting to receive something from httpd, then > > > you may have to look outside of Tomcat for the cause. > > > > > > - Chuck > > > > > > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > > > MATERIAL and is thus for use only by the intended recipient. If you > > > received > > > this in error, please contact the sender and delete the e-mail and its > > > attachments from all computers. > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > > > For additional commands, e-mail: users-h...@tomcat.apache.org > > > > > > > > > > > > > > > > > > _________________________________________________________________ > > Windows Live™: Keep your life in sync. > > http://windowslive.com/explore?ocid=TXT_TAGLM_BR_life_in_synch_052009 > > > > > > > > _________________________________________________________________ > Hotmail® goes with you. > http://windowslive.com/Tutorial/Hotmail/Mobile?ocid=TXT_TAGLM_WL_HM_Tutorial_Mobile1_052009 > > > _________________________________________________________________ Hotmail® has ever-growing storage! Don’t worry about storage limits. http://windowslive.com/Tutorial/Hotmail/Storage?ocid=TXT_TAGLM_WL_HM_Tutorial_Storage1_052009
