RE: mod_jk and Tomcat Lose Touch
This is what I thought, too, originally. However it wouldn't make sense since very few times do resources become *fewer* by default as software versions increase. With that in mind, and a hunch up my sleeve, I actually did some interesting testing. My test platform is my notebook, running RHL 9, 512MB RAM, Apache 1.3.27, JDK 1.4.1_03, and Tomcat 4.1.24. It's the exact same version of all software, including the kernel, that we're running on the server that's experiencing the problem. What I did was create 200 virtual hosts, 200 Tomcat instances running on 200 different ports, and tied them all together with mod_jk. I then created a test script which used 'lynx' to test a simple JSP on each virtual host and display the which JVM it was testing and what its results were. My results were *very* interesting. They are *reproducable* and *predictable*, which is at least better news than completely random! On the same software platform, where all defaults and limitations were exactly the same, the major affecting factors seem to be memory and swap, *not* any given resource within the OS itself. On the production system, this problem will not appear until it's running, say, 100 JVM's or so. The server has 2GB of RAM, 4GB of swap. On my notebook, however, it will appear with only 40 JVM's, which has 512MB of RAM and 1GB swap. Of course, when it's starting JVM's 30-40, it's doing a lot of swapping as resources are allocated. So how it turns out is as follows: If I start 1 JVM per minute, up to 40, it does just a little swapping per instance as it starts up. All JVM's up to #25 or so will work perfectly. Once you startup #26 and up, you start to lose a JVM here and there -- mod_jk will report its error and you'll never get a response. Essentially, mod_jk and the JVM lose touch. This will continue up to JVM #40 or so. Starting 1 per minute, I lost maybe 3-4 JVM's along the way. If I then start them back up, all the JVM's will work for a while, but eventually one or two will drop out again. Any more than 40 JVM's, and they all start dropping like flys. If I start all 40 JVM's 10 seconds apart, the swapping is tremendous by JVM #25, and *no* JVM will work at all after #25. Also, all the first 25 drop out as well. All of this behavior seems consistent with the behavior on the server, as well, only scaled-down. The server has 4x the RAM, so it doesn't exhibit this behavior until its memory gets full and it starts swapping processes out. I.e., if I start 100 JVM's on the server, it doesn't use any (or hardly any) swap space, and everything works perfectly. As the Web applications initiliaze, sessions are built, etc. eventually the server levels out at ~2GB of swap space being used, and this is when the problems start. This leads me to believe that the problem is with either memory not being properly allocated to mod_jk, even though there's plenty of swap available, *or* a timing issue where it tries to send a request to a JVM but for a split-second it doesn't get a response (due to paging, or whatever) and then perminently loses touch. This exact situation occurred a long time ago back in the days of JServ and mod_jserv. This is probably an issue for the developers to look at -- are there any mod_jk developers on this list? :-) I'm sure their knee-jerk reaction would be "upgrade to mod_jk2 and Apache2," but this is not a practical solution for me. Thanks for any help, and for reading so far! -Fred > Sounds to me like an OS resource issue. Total number of allowed open > connections, etc. Since it worked on 7.x and not on 9, I > would consider > that a default changed somewhere, somehow. > > John - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: mod_jk and Tomcat Lose Touch
Sounds to me like an OS resource issue. Total number of allowed open connections, etc. Since it worked on 7.x and not on 9, I would consider that a default changed somewhere, somehow. John Also, the same Apache 1.3.X and Tomcat 3.X and 4.X configurations worked perfectly on Red Hat 7.3, it was not until RH 9 that this began occuring. The behavior is consistent with Apache 1.3 and Apache 2 on RH 9. I've tried re-compiling and all the usual tricks. ... Do you use multiple Apaches on the server? - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: mod_jk and Tomcat Lose Touch
> Looks like there is a mismatch between the workers.properties and the > server.xml (Tomcat's config file). The server.xml sets up Tomcat. More > importantly, it sets up a Connector which opens a socket > connection on a > defined port (Look for the Connector' section in the Service > section of > the server.xml). This port is what mod_jk needs to know about in the > workers.properties. Some snippets from my config: ... > Check out both of these files and make sure the config is setup > cporrectly. It looks like either the Connector is not > configured or the > workers.properties does not have the right info for the > workers spawned > by Apache. Unfortunately it's not nearly this simple :-( Per my original post, we have ~150 Tomcat instances running, all properly configured, and 1 instance of Apache on the server. If I say, 'killall' the JVM's, and then start them back up again, everything will work perfectly. After a day or two, some of the sites will then lose their connectivity to Apache/JK. Simply restarting the JVM will "fix" things for a further day or two (for the sites configured to work with the JVM that I restart), thereby ruling-out a misconfiguration (though not ruling-out a possible non-optimal-configuration). Also, the same Apache 1.3.X and Tomcat 3.X and 4.X configurations worked perfectly on Red Hat 7.3, it was not until RH 9 that this began occuring. The behavior is consistent with Apache 1.3 and Apache 2 on RH 9. I've tried re-compiling and all the usual tricks. ... > Do you use multiple Apaches on the server? Just one. > Since I would be speculating wildly, why not post the > workers.properties > file as well as the Connector tags in the server.xml on the Tomcats. Well, 150 workers in the workers.properties file would be a bit large for starters ;-) So would 150 server.xml's... But regardless, I'm certain they're all configured properly, it's just a matter of Tomcat's AJP 1.3 service and mod_jk losing touch after some period of time. -Fred - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: mod_jk and Tomcat Lose Touch
On Tue, 2003-08-26 at 15:33, Fred Whipple wrote: > > In the settings you put in httpd.conf, you should have added a > > 'LogLevel' directive. If you did not, then you can! The line > > I have is: > > > > JkLogLevel error > > > > Change the 'error' to debug and restart Apache. > > This worked well, and yielded some errors: > > ** > [Tue Aug 26 16:25:19 2003] [jk_connect.c (203)]: jk_open_socket, connect() > failed errno = 111 > [Tue Aug 26 16:25:19 2003] [jk_ajp_common.c (626)]: Error connecting to > tomcat. Tomcat is probably not started or is listenning on the wrong port. > Failed errno = 111 Looks like there is a mismatch between the workers.properties and the server.xml (Tomcat's config file). The server.xml sets up Tomcat. More importantly, it sets up a Connector which opens a socket connection on a defined port (Look for the Connector' section in the Service section of the server.xml). This port is what mod_jk needs to know about in the workers.properties. Some snippets from my config: -Server.xml snippet- -End Server.xml snippet- -mod_jk's workers.properties file snippet- # # First tomcat server # worker.tomcat1.port=11009 worker.tomcat1.host=10.1.1.1 worker.tomcat1.type=ajp13 -End mod_jk's workers.properties file snippet- Check out both of these files and make sure the config is setup cporrectly. It looks like either the Connector is not configured or the workers.properties does not have the right info for the workers spawned by Apache. > All of these errors appeared at the same time I was trying to view a JSP on > a site that had "lost touch." I was doing a tail -f and the log file had > been quiet until I tried to view this JSP -- then all these popped-up at > about the same time. > > Immidiately I opened a telnet session to this particular Tomcat instance's > AJP 1.3 port, and verified it was responding. Also, *all* of the Tomcat > instances are on the same host as the Web server, so all the workers are > traveling over 127.0.0.1. Do you use multiple Apaches on the server? Since I would be speculating wildly, why not post the workers.properties file as well as the Connector tags in the server.xml on the Tomcats. Ben Ricker Wellinx.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: mod_jk and Tomcat Lose Touch
> In the settings you put in httpd.conf, you should have added a > 'LogLevel' directive. If you did not, then you can! The line > I have is: > > JkLogLevel error > > Change the 'error' to debug and restart Apache. This worked well, and yielded some errors: ** [Tue Aug 26 16:25:19 2003] [jk_connect.c (203)]: jk_open_socket, connect() failed errno = 111 [Tue Aug 26 16:25:19 2003] [jk_ajp_common.c (626)]: Error connecting to tomcat. Tomcat is probably not started or is listenning on the wrong port. Failed errno = 111 [Tue Aug 26 16:25:19 2003] [jk_ajp_common.c (874)]: Error connecting to the Tomcat process. [Tue Aug 26 16:25:19 2003] [jk_ajp_common.c (1190)]: sending request to tomcat failed in send loop. err=0 [Tue Aug 26 16:25:19 2003] [jk_connect.c (203)]: jk_open_socket, connect() failed errno = 111 [Tue Aug 26 16:25:19 2003] [jk_ajp_common.c (626)]: Error connecting to tomcat. Tomcat is probably not started or is listenning on the wrong port. Failed errno = 111 [Tue Aug 26 16:25:19 2003] [jk_ajp_common.c (874)]: Error connecting to the Tomcat process. [Tue Aug 26 16:25:19 2003] [jk_ajp_common.c (1190)]: sending request to tomcat failed in send loop. err=1 [Tue Aug 26 16:25:19 2003] [jk_connect.c (203)]: jk_open_socket, connect() failed errno = 111 [Tue Aug 26 16:25:19 2003] [jk_ajp_common.c (626)]: Error connecting to tomcat. Tomcat is probably not started or is listenning on the wrong port. Failed errno = 111 [Tue Aug 26 16:25:19 2003] [jk_ajp_common.c (874)]: Error connecting to the Tomcat process. [Tue Aug 26 16:25:19 2003] [jk_ajp_common.c (1190)]: sending request to tomcat failed in send loop. err=2 [Tue Aug 26 16:25:19 2003] [jk_ajp_common.c (1198)]: Error connecting to tomcat. Tomcat is probably not started or is listenning on the wrong port. Failed errno = 111 [Tue Aug 26 16:25:19 2003] [jk_ajp_common.c (970)]: ERROR sending data to client. Connection aborted or network problems [Tue Aug 26 16:25:50 2003] [jk_ajp_common.c (970)]: ERROR sending data to client. Connection aborted or network problems [Tue Aug 26 16:26:04 2003] [jk_ajp_common.c (970)]: ERROR sending data to client. Connection aborted or network problems [Tue Aug 26 16:26:15 2003] [jk_ajp_common.c (970)]: ERROR sending data to client. Connection aborted or network problems [Tue Aug 26 16:26:22 2003] [jk_ajp_common.c (970)]: ERROR sending data to client. Connection aborted or network problems ** All of these errors appeared at the same time I was trying to view a JSP on a site that had "lost touch." I was doing a tail -f and the log file had been quiet until I tried to view this JSP -- then all these popped-up at about the same time. Immidiately I opened a telnet session to this particular Tomcat instance's AJP 1.3 port, and verified it was responding. Also, *all* of the Tomcat instances are on the same host as the Web server, so all the workers are traveling over 127.0.0.1. Any ideas? Thanks! -Fred - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: mod_jk and Tomcat Lose Touch
On Mon, 2003-08-25 at 13:08, Fred Whipple wrote: > Ah, the symptoms, of course ;-) > > There actually aren't any errors AFAIK. Perhaps there's a more verbose > logging level? I couldn't find any obvious info on this in the manual. In the settings you put in httpd.conf, you should have added a 'LogLevel' directive. If you did not, then you can! The line I have is: JkLogLevel error Change the 'error' to debug and restart Apache. Ben Ricker Wellinx, Inc. > What actually happens is if I make a request to a JSP, for example, that > should normally be forwarded onto Tomcat, the request simply hangs forever. > Apache logs the request nothing in the Tomcat logs suggest to me it's > receiving the request. > > Thanks, > > -Fred > > > -Original Message- > > From: Ben Ricker [mailto:[EMAIL PROTECTED] > > Sent: Monday, August 25, 2003 1:09 PM > > To: [EMAIL PROTECTED] > > Subject: Re: mod_jk and Tomcat Lose Touch > > > > > > What kind of error are you getting that makes you thing they > > are "losing > > touch". Check the mod_jk log and see what kind of errors it > > is throwing > > and post an example. > > > > Ben Ricker > > Wellinx, Inc. > > > - > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: mod_jk and Tomcat Lose Touch
Ah, the symptoms, of course ;-) There actually aren't any errors AFAIK. Perhaps there's a more verbose logging level? I couldn't find any obvious info on this in the manual. What actually happens is if I make a request to a JSP, for example, that should normally be forwarded onto Tomcat, the request simply hangs forever. Apache logs the request nothing in the Tomcat logs suggest to me it's receiving the request. Thanks, -Fred > -Original Message- > From: Ben Ricker [mailto:[EMAIL PROTECTED] > Sent: Monday, August 25, 2003 1:09 PM > To: [EMAIL PROTECTED] > Subject: Re: mod_jk and Tomcat Lose Touch > > > What kind of error are you getting that makes you thing they > are "losing > touch". Check the mod_jk log and see what kind of errors it > is throwing > and post an example. > > Ben Ricker > Wellinx, Inc. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: mod_jk and Tomcat Lose Touch
What kind of error are you getting that makes you thing they are "losing touch". Check the mod_jk log and see what kind of errors it is throwing and post an example. Ben Ricker Wellinx, Inc. On Mon, 2003-08-25 at 10:54, Fred Whipple wrote: > Hi all, > > On one of our servers, we were running Red Hat Linux 7.3 with Apache > 1.3.X, mod_jk, Sun JDK 1.4.1, and many (~150) instances Tomcat of > various flavors. Everything worked perfectly until we upgraded to Red > Hat 9. Now, it seems like mod_jk and Tomcat are losing touch until the > Tomcat JVM is restarted. This happens after maybe a day or so using > both Apache 1.3.X and Apache 2.0.X. > > >From what we've observed, it seems to a problem specific to mod_jk. > This is because running Apache 1.3.X or 2.0.X under Red Hat 9 with the > old-skool mod_jserv and ~100 Tomcat instances does not cause a problem > at all. Also, running either Apache 2 or 1.3 on RH 9 with just a couple > JVM's does not cause a problem either. > > I've already tried re-compiling mod_jk on RH9, the same server that's > having the trouble to account for any library change issues. No dice > I'm afraid. > > Any ideas? > > Thanks! > > -Fred > > > - > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]