Re: Tomcat 4.1.12 hanging under stress?
Ben Ricker wrote: On Mon, 2002-10-28 at 20:24, William Lee wrote: Hmm, it may have something to do with the kernel, I'll try later to see whether this is the case. I don't think the log gives me much though. Anyhow, I've made a weird discovery that I found interesting. After the other info you gave me, I am less concerned with the kernel now. I would still upgrade it, though. There is a nasty networking bug in the 2,4,9 kernel, I believe, that effects scalability. It seems like there's a barrier when the JVM consume around 90MB of RAM. Once the JVM got over that barrier, then things will keep working. Well, it turned out this problem may not have anything to do with the JVM at all. After trying various different JREs, -X flags, and garbage collection info, etc, I updated my tomcat to 4.1.14 (test) and now I got the following exception in stdout when I got into the situation where the connection hung: Class org.apache.commons.logging.impl.Log4JCategoryLog does not implement Log at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:555) at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:289) at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:409) at org.apache.tomcat.util.log.CommonLogHandler.log(CommonLogHandler.java:97) at org.apache.tomcat.util.log.Log.log(Log.java:198) at org.apache.tomcat.util.log.Log.log(Log.java:192) at org.apache.tomcat.util.log.Log.log(Log.java:165) at org.apache.tomcat.util.threads.ThreadPool.logFull(ThreadPool.java:260) at org.apache.tomcat.util.threads.ThreadPool.runIt(ThreadPool.java:226) at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:509) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:533) at java.lang.Thread.run(Thread.java:512) Looking into the 4.1.14 source code it turned out that the thread (maybe the connection thread?) just died when calling the function logFull(), and somehow log4j failed to do so (?). Does this have anything to do with class loader and using different versions of log4j? Does anyone know how tomcat logged its messages? -- William Lee (Will)| Sendmail Inc. Email: [EMAIL PROTECTED] | http://www.sendmail.com Tel:(510) 594-5505| -- To unsubscribe, e-mail: mailto:tomcat-user-unsubscribe;jakarta.apache.org For additional commands, e-mail: mailto:tomcat-user-help;jakarta.apache.org
Re: Tomcat 4.1.12 hanging under stress?
On Mon, 2002-10-28 at 20:24, William Lee wrote: Hmm, it may have something to do with the kernel, I'll try later to see whether this is the case. I don't think the log gives me much though. Anyhow, I've made a weird discovery that I found interesting. After the other info you gave me, I am less concerned with the kernel now. I would still upgrade it, though. There is a nasty networking bug in the 2,4,9 kernel, I believe, that effects scalability. It seems like there's a barrier when the JVM consume around 90MB of RAM. Once the JVM got over that barrier, then things will keep working. You may be dealing with garbage collection issues. If the GC gets going before that, it will lock everything up while it GCs. Try messing with the following options: -Xms and -Xmx using CATALINA_OPTS set in the startup script. For example: CATALINA_OPTS=-Xms128M -Xmx2048M would set the heap size minimum to 128meg and would force garbage collection at 2048meg. Play with that and see how it affects the slowdown. You can alos set two other options to help out debugging: -Xincgc and -Xloggc:gcinfo.log I forgot what the first setting does (ingrease GC?) but the second one turns on logging when it garbage collects. If you watch that file as the failure happens, you can tell if garbage collecting is having an effect or not. HTH, Ben Ricker Wellinx, Inc. -- To unsubscribe, e-mail: mailto:tomcat-user-unsubscribe;jakarta.apache.org For additional commands, e-mail: mailto:tomcat-user-help;jakarta.apache.org
Tomcat 4.1.12 hanging under stress?
I'm running a Tomcat 4.1.12 (with the coyote connectors running SSL) on RH Linux with a 2.4.9 kernel and IBM JRE 1.3.1 build cxia32131-20020622. I initiated around 50 concurrent https connections from JMeter to the test server I have (which is a moderately powerful dual PIII 500 with 512 MB RAM and Ultra-SCSI drives) and I realized some random hangings after running for a while. When the hanging happened, I could still connect to the port that I opened. However, I couldn't do anything once I connected to the port. The JVM seemed to be running still, but tomcat wouldn't response. BTW, I tried connecting to the port using openssl s_client ... and I was able to connect to the port when it hung, but I can't issue any HTTP commands. I can, however, shutdown the server from the control port with no problem. Can somebody tell me how to address / debug this problem? -- William Lee (Will)| Sendmail Inc. Email: [EMAIL PROTECTED] | http://www.sendmail.com Tel:(510) 594-5505| -- To unsubscribe, e-mail: mailto:tomcat-user-unsubscribe;jakarta.apache.org For additional commands, e-mail: mailto:tomcat-user-help;jakarta.apache.org
Re: Tomcat 4.1.12 hanging under stress?
On Mon, 2002-10-28 at 14:38, William Lee wrote: I'm running a Tomcat 4.1.12 (with the coyote connectors running SSL) on RH Linux with a 2.4.9 kernel and IBM JRE 1.3.1 build cxia32131-20020622. I am wondering about the kernel version here; you have rather old kernel, a version which is considered unstable. I would suggest upgrading the kernel, especially since you have SMP. This is just a small suggestion.anything in the logs? Ben Ricker Wellin.com I initiated around 50 concurrent https connections from JMeter to the test server I have (which is a moderately powerful dual PIII 500 with 512 MB RAM and Ultra-SCSI drives) and I realized some random hangings after running for a while. When the hanging happened, I could still connect to the port that I opened. However, I couldn't do anything once I connected to the port. The JVM seemed to be running still, but tomcat wouldn't response. BTW, I tried connecting to the port using openssl s_client ... and I was able to connect to the port when it hung, but I can't issue any HTTP commands. I can, however, shutdown the server from the control port with no problem. Can somebody tell me how to address / debug this problem? -- William Lee (Will)| Sendmail Inc. Email: [EMAIL PROTECTED] | http://www.sendmail.com Tel:(510) 594-5505| -- To unsubscribe, e-mail: mailto:tomcat-user-unsubscribe;jakarta.apache.org For additional commands, e-mail: mailto:tomcat-user-help;jakarta.apache.org -- To unsubscribe, e-mail: mailto:tomcat-user-unsubscribe;jakarta.apache.org For additional commands, e-mail: mailto:tomcat-user-help;jakarta.apache.org
Re: Tomcat 4.1.12 hanging under stress?
Ben Ricker wrote: On Mon, 2002-10-28 at 14:38, William Lee wrote: I'm running a Tomcat 4.1.12 (with the coyote connectors running SSL) on RH Linux with a 2.4.9 kernel and IBM JRE 1.3.1 build cxia32131-20020622. I am wondering about the kernel version here; you have rather old kernel, a version which is considered unstable. I would suggest upgrading the kernel, especially since you have SMP. This is just a small suggestion.anything in the logs? Ben Ricker Wellin.com Hmm, it may have something to do with the kernel, I'll try later to see whether this is the case. I don't think the log gives me much though. Anyhow, I've made a weird discovery that I found interesting. It seems like there's a barrier when the JVM consume around 90MB of RAM. Once the JVM got over that barrier, then things will keep working. The strange thing is, under not-so-heavy load ( 30 concurrent users), somehow it failed to break that barrier and the socket hung when it reaches that point. If I drive the load up ( 50, 100, or even 200 concurrent connections), somehow it breaks that barrier and things will work without any problem. Does anyone know why this is the case? -- William Lee (Will)| Sendmail Inc. Email: [EMAIL PROTECTED] | http://www.sendmail.com Tel:(510) 594-5505| -- To unsubscribe, e-mail: mailto:tomcat-user-unsubscribe;jakarta.apache.org For additional commands, e-mail: mailto:tomcat-user-help;jakarta.apache.org
