mod_jk hangs
Just yesterday we started running into problems requesting pages on our production website through mod_jk. Previously everything had been working fine for months. Essentially what happens is when you request the page for the first time (doesn't seem to matter what browser you're using) the browser just sits and spins. Then after hitting refresh (Ctrl+R) a couple times it finally comes up. Subsequent visits to the same page continue to work for me in Mozilla but after closing IE and reopening it after a couple more requests it hangs again. In case it helps, the site in question is http://www.atmreports.com. I simply cannot figure out what is going on. I do see many errors in the mod_jk.log, but these had been there before when everything worked fine too: [Wed May 05 09:27:46 2004] [jk_ajp_common.c (1137)]: Error reading reply from tomcat. Tomcat is down or network problems. [Wed May 05 09:27:46 2004] [jk_ajp_common.c (1290)]: ERROR: Receiving from tomcat failed, recoverable operation. err=0 [Wed May 05 09:27:46 2004] [jk_ajp_common.c (1309)]: sending request to tomcat failed in send loop. err=0 [Wed May 05 09:27:46 2004] [jk_ajp_common.c (738)]: ERROR: can't receive the response message from tomcat, network problems or tomcat is down. err=-1 A few things to note, any of which may contribute to the problem but for the life of me I can't figure out how: 1. On Friday we just moved out datacenter to a new location. However we see this problem even internally at the location so I don't believe it is any sort of routing issue (and the problem didn't start until Tuesday). 2. The clock on this webserver was many hours off. On Tuesday (yes, the day the problems started) the clock was synchronized and the timezone correctly set so the machine now has correct time. This seems to be the most probable cause simply because it happened around the same time, but I just can't understand why or how. 3. Yesterday we rebooted the box to see if that would help. It did. We didn't have any more problems at all that day until this morning we see the problems again. Today we recycled just tomcat and that seems to have fixed it as well, though I expect that by tomorrow I will see the problem again. 4. I wouldn't think this could have anything to do with that new Sasser worm, but listing that just in case. The machine is running Linux kernel 2.4.20-8 with httpd-2.0.40-21 and jakarta-tomcat-5.0.18. We are running 4 instances of tomcat on the box, each started with the following JVM options: -Xmn128M -Xmx1024M -Djava.awt.headless=true Nothing having to do with configuration has changed at all in a long time. Here is the relevant configuration sections: # Mod_jk (Tomcat connector) LoadModule jk_module modules/mod_jk.so IfModule mod_jk.c JkWorkersFile /etc/httpd/conf/workers.properties JkLogFile /var/log/httpd/mod_jk.log JkLogLevel info JkLogStampFormat [%a %b %d %H:%M:%S %Y] JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories JkRequestLogFormat %w %V %T /IfModule ...and later (in a VirtualHost section)... JKMount /myreports balancer JKMount /myreports/* balancer Here is our workers.properties file (comments removed): workers.java_home=/usr/java/j2sdk1.4.2_03 ps=/ worker.list=balancer worker.ajp13_0.port=8090 worker.ajp13_0.host=localhost worker.ajp13_0.type=ajp13 worker.ajp13_0.lbfactor=1 worker.ajp13_1.port=8091 worker.ajp13_1.host=localhost worker.ajp13_1.type=ajp13 worker.ajp13_1.lbfactor=1 worker.ajp13_2.port=8092 worker.ajp13_2.host=localhost worker.ajp13_2.type=ajp13 worker.ajp13_2.lbfactor=1 worker.ajp13_3.port=8093 worker.ajp13_3.host=localhost worker.ajp13_3.type=ajp13 worker.ajp13_3.lbfactor=1 worker.balancer.type=lb worker.balancer.balanced_workers=ajp13_0, ajp13_1, ajp13_2, ajp13_3 worker.balancer.sticky_session=1 Any help would be greatly appreciated! Thanks, -shawn -- Shawn Wilson [EMAIL PROTECTED] Software Developer, ATMReports.com PH: 877-327-0873, FAX: 406-294-5806 smime.p7s Description: S/MIME Cryptographic Signature
Re: mod_jk hangs
Okay, here is the section from our server.xml in production: Service name=Catalina Connector port=8080 / !-- This is here for compatibility only, not required -- Connector port=8090 protocol=AJP/1.3 / Engine name=Catalina defaultHost=localhost jvmRoute=ajp13_0 Logger className=org.apache.catalina.logger.FileLogger / Realm className=org.apache.catalina.realm.UserDatabaseRealm resourceName=UserDatabase / Host name=localhost appBase=webapps / /Engine /Service I thought it interesting the comment that this is here for compatibility only, not required. It sounds to me that connector is required if I want AJP/1.3 isn't it? I can't remember if that comment was there or if I (mistakingly) added it. Anyways, I see that I could probably do well to add the minSpareThreads and maxSpareThreads parameters. Any idea where I should look in my httpd.conf file to see what values to set these to? Thanks, -shawn Randall Svancara wrote: You might want to check to make sure you have enough tomcat threads for each apache thread. I usually give tomcat a few more threads than apache just to make sure I do not run out. ---snip !-- Define a Coyote/JK2 AJP 1.3 Connector on port 8009 -- Connector port=8009 enableLookups=false redirectPort=8443 debug=0 minSpareThreads=100 maxSpareThreads=300 protocol=AJP/1.3 / ---snip So you might want to increase min and max spare threads to accomodate more requestes from apache to tomcat via mod_jk. Randall -Original Message- From: Shawn Wilson [mailto:[EMAIL PROTECTED] Sent: Wednesday, May 05, 2004 9:50 AM To: [EMAIL PROTECTED] Subject: mod_jk hangs Just yesterday we started running into problems requesting pages on our production website through mod_jk. Previously everything had been working fine for months. Essentially what happens is when you request the page for the first time (doesn't seem to matter what browser you're using) the browser just sits and spins. Then after hitting refresh (Ctrl+R) a couple times it finally comes up. Subsequent visits to the same page continue to work for me in Mozilla but after closing IE and reopening it after a couple more requests it hangs again. In case it helps, the site in question is http://www.atmreports.com. I simply cannot figure out what is going on. I do see many errors in the mod_jk.log, but these had been there before when everything worked fine too: [Wed May 05 09:27:46 2004] [jk_ajp_common.c (1137)]: Error reading reply from tomcat. Tomcat is down or network problems. [Wed May 05 09:27:46 2004] [jk_ajp_common.c (1290)]: ERROR: Receiving from tomcat failed, recoverable operation. err=0 [Wed May 05 09:27:46 2004] [jk_ajp_common.c (1309)]: sending request to tomcat failed in send loop. err=0 [Wed May 05 09:27:46 2004] [jk_ajp_common.c (738)]: ERROR: can't receive the response message from tomcat, network problems or tomcat is down. err=-1 A few things to note, any of which may contribute to the problem but for the life of me I can't figure out how: 1. On Friday we just moved out datacenter to a new location. However we see this problem even internally at the location so I don't believe it is any sort of routing issue (and the problem didn't start until Tuesday). 2. The clock on this webserver was many hours off. On Tuesday (yes, the day the problems started) the clock was synchronized and the timezone correctly set so the machine now has correct time. This seems to be the most probable cause simply because it happened around the same time, but I just can't understand why or how. 3. Yesterday we rebooted the box to see if that would help. It did. We didn't have any more problems at all that day until this morning we see the problems again. Today we recycled just tomcat and that seems to have fixed it as well, though I expect that by tomorrow I will see the problem again. 4. I wouldn't think this could have anything to do with that new Sasser worm, but listing that just in case. The machine is running Linux kernel 2.4.20-8 with httpd-2.0.40-21 and jakarta-tomcat-5.0.18. We are running 4 instances of tomcat on the box, each started with the following JVM options: -Xmn128M -Xmx1024M -Djava.awt.headless=true Nothing having to do with configuration has changed at all in a long time. Here is the relevant configuration sections: # Mod_jk (Tomcat connector) LoadModule jk_module modules/mod_jk.so IfModule mod_jk.c JkWorkersFile /etc/httpd/conf/workers.properties JkLogFile /var/log/httpd/mod_jk.log JkLogLevel info JkLogStampFormat [%a %b %d %H:%M:%S %Y] JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories JkRequestLogFormat %w %V %T /IfModule ...and later (in a VirtualHost section)... JKMount /myreports balancer JKMount /myreports/* balancer Here is our workers.properties file (comments
Re: mod_jk hangs
Looks like the maximum for everyting in my config was 150 so I'll try setting maxSpareThreads for Tomcat to 150 as well. If this is indeed the cause of the problem, any idea why it would just start happening yesterday after months without problem? My only guess is that perhaps we just happened to reach some threshold of traffic yesterday that we never did before. Actually, that may indeed be the case since we usually get a surge of traffic during the beginning of a month. Thanks, -shawn Randall Svancara wrote: Well, that depends on what platform and how apache was compiled. As I understand on windows, apache uses the mpm modules and unix it uses forking or it can use mpm. You need to find the section for you platform and make the ajustments as neccessary. Below is the section where you need to look in your apache configuration file, usually called httpd.conf, but i have seen them named apache.conf (gentoo anyone). But you can see the min and max spare servers/threads listed in each section. Just adjust accordingly. I hope this helps. IfModule prefork.c StartServers 75 MinSpareServers 75 MaxSpareServers 300 MaxClients 300 MaxRequestsPerChild 0 /IfModule # worker MPM # StartServers: initial number of server processes to start # MaxClients: maximum number of simultaneous client connections # MinSpareThreads: minimum number of worker threads which are kept spare # MaxSpareThreads: maximum number of worker threads which are kept spare # ThreadsPerChild: constant number of worker threads in each server process # MaxRequestsPerChild: maximum number of requests a server process serves IfModule worker.c StartServers 75 MaxClients 300 MinSpareThreads 75 MaxSpareThreads 300 ThreadsPerChild 25 MaxRequestsPerChild 0 /IfModule # perchild MPM # NumServers: constant number of server processes # StartThreads: initial number of worker threads in each server process # MinSpareThreads: minimum number of worker threads which are kept spare # MaxSpareThreads: maximum number of worker threads which are kept spare # MaxThreadsPerChild: maximum number of worker threads in each server process # MaxRequestsPerChild: maximum number of connections per server process IfModule perchild.c NumServers 75 StartThreads 75 MinSpareThreads 75 MaxSpareThreads 300 MaxThreadsPerChild 20 MaxRequestsPerChild 0 /IfModule # WinNT MPM # ThreadsPerChild: constant number of worker threads in the server process # MaxRequestsPerChild: maximum number of requests a server process serves IfModule mpm_winnt.c ThreadsPerChild 250 MaxRequestsPerChild 0 /IfModule -Original Message- From: Shawn Wilson [mailto:[EMAIL PROTECTED] Sent: Wednesday, May 05, 2004 10:21 AM To: Tomcat Users List Subject: Re: mod_jk hangs Okay, here is the section from our server.xml in production: Service name=Catalina Connector port=8080 / !-- This is here for compatibility only, not required -- Connector port=8090 protocol=AJP/1.3 / Engine name=Catalina defaultHost=localhost jvmRoute=ajp13_0 Logger className=org.apache.catalina.logger.FileLogger / Realm className=org.apache.catalina.realm.UserDatabaseRealm resourceName=UserDatabase / Host name=localhost appBase=webapps / /Engine /Service I thought it interesting the comment that this is here for compatibility only, not required. It sounds to me that connector is required if I want AJP/1.3 isn't it? I can't remember if that comment was there or if I (mistakingly) added it. Anyways, I see that I could probably do well to add the minSpareThreads and maxSpareThreads parameters. Any idea where I should look in my httpd.conf file to see what values to set these to? Thanks, -shawn Randall Svancara wrote: You might want to check to make sure you have enough tomcat threads for each apache thread. I usually give tomcat a few more threads than apache just to make sure I do not run out. ---snip !-- Define a Coyote/JK2 AJP 1.3 Connector on port 8009 -- Connector port=8009 enableLookups=false redirectPort=8443 debug=0 minSpareThreads=100 maxSpareThreads=300 protocol=AJP/1.3 / ---snip So you might want to increase min and max spare threads to accomodate more requestes from apache to tomcat via mod_jk. Randall -Original Message- From: Shawn Wilson [mailto:[EMAIL PROTECTED] Sent: Wednesday, May 05, 2004 9:50 AM To: [EMAIL PROTECTED] Subject: mod_jk hangs Just yesterday we started running into problems requesting pages on our production website through mod_jk. Previously everything had been working fine for months. Essentially what happens is when you request the page for the first time (doesn't seem to matter what browser you're using) the browser just sits and spins. Then after hitting refresh (Ctrl+R) a couple times it finally comes up
Re: Admin Webapp Problems (WAS: MBean server?)
Okay, this was just dumb on my part. When I installed Tomcat I installed the tomcat-noexamples and tomcat-admin RPM's. However, I noticed that installing Tomcat on a different machine (with examples), the first section in the server.xml file having to do with (believe it or not!) MBean server was not commented out like it was in the noexamples server.xml file. So basically that's all the problem was... if I had looked at server.xml more closely sooner I could have saved myself lots of frustration! Thanks for all the help, -shawn Jacob Kjome wrote: Hi Shawn, I'd recommend installing the full non-RPM version that includes all necessary jars. In that version, you will see the following files in common/endorsed: xercesImpl.jar xmlParserAPIs.jar What Bill says it correct, though. The Xerces version that come with Tomcat causes problems with Struts apps because of an XML parsing bug in Xerces which was recently fixed. You can either do as he says and go back to using 2.1.0 version or move to the latest jars which contain the fix: http://gump.covalent.net/jars/latest/xml-xerces2/ Otherwise, you could also check the xml commons to grab a xml-commons-api's jar that contains an earlier version of the xerces packages equivalent to the 2.1.0 release. http://xml.apache.org/commons/ Jake Tuesday, November 26, 2002, 10:05:27 AM, you wrote: SW Hi Bill, SW Thanks for the info. I checked, but there isn't any xerces jar in SW $CATALINA_HOME/common/endorsed... only jaxp_parser_impl.jar and SW xml-commons-apis.jar. This was just a basic RPM install of Tomcat along SW with the admin webapps. Also, the exception trace I posted _was_ from SW catalina.out, so that's as detailed as it gets. SW By the way, what is this MBean server it is looking for anyways? SW Thanks, SW -shawn SW Bill Barker wrote: I'm guessing that there is a more informative message in 'catalina.out', and that the problem is that 4.1.12 doesn't work with xerces-2.2.x (other than 'nightly'). Replace the xerces jars in $CATALINA_HOME/common/endorsed with the 2.1.0 version, and it should be fine. Shawn Wilson [EMAIL PROTECTED] wrote in message [EMAIL PROTECTED]">news:[EMAIL PROTECTED]... I have the following packages installed on a RedHat 7.3 box: tomcat4-4.1.12-full.2jpp tomcat4-admin-webapps-4.1.12-full.2jpp Tomcat starts up fine and I can log into the admin app without problem. However, the moment I try to do anything in the admin app I get a HTTP Status 503 - Servlet action is currently unavailable error, and I see this in my catalina.out log file: javax.servlet.UnavailableException: MBeanServer is not available at org.apache.webapp.admin.ApplicationServlet.initServer(Unknown Source) at org.apache.webapp.admin.ApplicationServlet.getServer(Unknown Source) at org.apache.webapp.admin.TomcatTreeBuilder.buildTree(Unknown Source) at org.apache.webapp.admin.SetUpTreeAction.perform(Unknown Source) at org.apache.struts.action.ActionServlet.processActionPerform(ActionServlet.ja va) at org.apache.struts.action.ActionServlet.process(ActionServlet.java) at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java) at javax.servlet.http.HttpServlet.service(HttpServlet.java) at javax.servlet.http.HttpServlet.service(HttpServlet.java) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Unknown Source) ... ... What does this exception mean, and what do I need to do to get the admin app running? Thanks, -shawn -- ==== Shawn Wilson [[EMAIL PROTECTED]] Software Developer, ATMReports.com PH: 877-327-0873, FAX: 406-294-5806 -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED] -- ==== Shawn Wilson [[EMAIL PROTECTED]] Software Developer, ATMReports.com PH: 877-327-0873, FAX: 406-294-5806 -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
MBean server?
I have the following packages installed on a RedHat 7.3 box: tomcat4-4.1.12-full.2jpp tomcat4-admin-webapps-4.1.12-full.2jpp Tomcat starts up fine and I can log into the admin app without problem. However, the moment I try to do anything in the admin app I get a HTTP Status 503 - Servlet action is currently unavailable error, and I see this in my catalina.out log file: javax.servlet.UnavailableException: MBeanServer is not available at org.apache.webapp.admin.ApplicationServlet.initServer(Unknown Source) at org.apache.webapp.admin.ApplicationServlet.getServer(Unknown Source) at org.apache.webapp.admin.TomcatTreeBuilder.buildTree(Unknown Source) at org.apache.webapp.admin.SetUpTreeAction.perform(Unknown Source) at org.apache.struts.action.ActionServlet.processActionPerform(ActionServlet.java) at org.apache.struts.action.ActionServlet.process(ActionServlet.java) at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java) at javax.servlet.http.HttpServlet.service(HttpServlet.java) at javax.servlet.http.HttpServlet.service(HttpServlet.java) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Unknown Source) ... ... What does this exception mean, and what do I need to do to get the admin app running? Thanks, -shawn -- Shawn Wilson [[EMAIL PROTECTED]] Software Developer, ATMReports.com PH: 877-327-0873, FAX: 406-294-5806 -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
RE: [OT] Job Openings
I'm really not sure what your post has to do with Tomcat. Perhaps you should post to a jobs list? -shawn Shawn Wilson [[EMAIL PROTECTED]] Software Developer, ATMReports.com PH: 877-327-0873, FAX: 406-294-5806 -Original Message- From: Pooleery, Manoj [mailto:[EMAIL PROTECTED]] Sent: Wednesday, September 18, 2002 8:21 AM To: 'Tomcat Users List' Subject: Job Openings Hi, My company is looking for experienced Java professionals with 3-4 years of experience. Experience in core, serverside Java, JSP, XML and XSL is a must. If you are interested, please send your resume to [EMAIL PROTECTED] Thanks Manoj. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] Sent: Wednesday, September 18, 2002 11:03 AM To: Tomcat Users List Subject: RE: Redirect not working I'm quite sure: in one case I explicitly do a sendRedirect(newurl) from within a servlet. Furthermore, the response code is 302, which is a temporary redirection. Jonathan Soons [EMAIL PROTECTED] on 18/09/2002 15.45.03 Please respond to Tomcat Users List [EMAIL PROTECTED] To:Tomcat Users List [EMAIL PROTECTED] cc: Subject:RE: Redirect not working Are you sure that is a Redirect? Redirect requires /path - URL. Yours looks like an Alias. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] Sent: Wednesday, September 18, 2002 7:12 AM To: [EMAIL PROTECTED] Subject: Redirect not working Hi all. My configuration is: Apache 2.0.35 and Tomcat 4.0.3 (or 4.0.4, same behaviour), with mod_webapp and warp connector. Solaris 8, jdk 1.4.0. If I deploy examples webapp and access it through Tomcat's standalone listener (port 8080) everything works ok. When I access it through Apache, it works well until I find some kind of redirection (ie /examples/ - /examples/index.html), then I get a page with the message: Apache Tomcat/4.0.3 -HTTP Status 302 - Moved Temporarily but redirection is NOT followed by the browser (I've tried various versions of IE, Netscape, Mozilla...). If I try to connect using telnet, the response with port 8080 is: HTTP/1.1 302 Moved Temporarily while through Apache is: HTTP/1.1 Moved Temporarily Nevertheless, in access.log of Apache the status code for my request is correctly set to 302. Same behaviour if I do a sendRedirect from within a servlet or jsp, redirecting to another servlet (no static content at all). Can anybody help me? TIA, Carlo -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED] -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED] Carlo Montanari Sysadm Unix office T-Systems Italia debis IT Services Italia S.p.A. Via degli Ontani, 25 36100 - Vicenza Phone: +39 0444 558355 Fax: +39 0444 558352 Mobile: +39 348 4530249 e-mail: [EMAIL PROTECTED] Internet: http://www.t-systems.it -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED] -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
Removing files from /lib
I must not be understanding completely the way the classloaders work under Tomcat. I thought that if I stop a webapp through the manager application that should unload its jar files. However, even after it's stopped, I find that log4j jar file is locked so I can't remove it. Why would that one file be locked but the others not? -shawn Shawn Wilson [[EMAIL PROTECTED]] Software Developer, ATMReports.com PH: 877-327-0873, FAX: 406-294-5806 -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
RE: Removing files from /lib
It is in WEB-INF/lib, which is why I thought it would be unlocked when I stop that webapp. -shawn Shawn Wilson [[EMAIL PROTECTED]] Software Developer, ATMReports.com PH: 877-327-0873, FAX: 406-294-5806 -Original Message- From: Andrew Conrad [mailto:[EMAIL PROTECTED]] Sent: Wednesday, August 28, 2002 10:44 AM To: 'Tomcat Users List' Subject: RE: Removing files from /lib Where are you keeping log4j? Is it in common/lib or some place such as that or is it stored in you WEB-INF/lib folder? From my understanding if it's in common or shared, then unloading the webapp won't release the files, since webapps inherit common and shared classloaders, they don't actually create them. - Andrew -Original Message- From: Shawn Wilson [mailto:[EMAIL PROTECTED]] Sent: Wednesday, August 28, 2002 12:05 PM To: tomcat Subject: Removing files from /lib I must not be understanding completely the way the classloaders work under Tomcat. I thought that if I stop a webapp through the manager application that should unload its jar files. However, even after it's stopped, I find that log4j jar file is locked so I can't remove it. Why would that one file be locked but the others not? -shawn Shawn Wilson [[EMAIL PROTECTED]] Software Developer, ATMReports.com PH: 877-327-0873, FAX: 406-294-5806 -- To unsubscribe, e-mail: mailto:tomcat-user- [EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED] -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED] -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
Tomcat memory usage
Here's are tomcat version: Apache Tomcat/4.0.4-b3 We've had it running for a while now and I just noticed that it is using a lot of memory! 142MB physical and 600MB virtual. I'm wondering if there is any way I can tell what webapp is consuming this memory? I've disabled all webapps but /manager and two others, but the memory usage did not go down. Thanks, -shawn Shawn Wilson [[EMAIL PROTECTED]] Software Developer, ATMReports.com PH: 877-327-0873, FAX: 406-294-5806 -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
