Hello. I'm running Tomcat/4.1.27 on a Debian Woody system with a vanilla Linux 2.4.21 kernel compiled with gcc 2.95.4 20011002 (Debian prerelease); "java -version" produces: java version "1.4.2" Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2-b28) Java HotSpot(TM) Client VM (build 1.4.2-b28, mixed mode)
While stress-testing one of our web app, I noticed that after a while, Tomcat stops responding to HTTP requests. Basically, I'm running the following short shell script: for i in `seq 1 1000`; do curl -d "variables_to_the_servlet_causing_heavy_operations" http://server:10003/mywebapp/myservlet & done This specific POST causes the servlet to perform some relatively "heavy" operations inside a block that is synchronized() on a common object. I've configured tomcat with the following connection in server.xml: <!-- Define a non-SSL HTTP/1.1 Connector on port 10003 --> <Connector className="org.apache.catalina.connector.http.HttpConnector" port="10003" minProcessors="5" maxProcessors="2000" enableLookups="true" redirectPort="8443" acceptCount="10" debug="100" connectionTimeout="60000"/> A lot of HttpProcessors are created, and they queue up waiting to get the synchronization on the common object. After waiting for a long while, I kill all remaining curl processes. For several minutes the servlet keeps logging its handling of the incoming requests, but it seems to slow down. When everything quiets down, tomcat seems to be unable to service any http requests. Any attempt to connect to port 10003 with a web browser eventually causes a "No data in response"-style error message. At this point it seems the http connector listening on port 10003 has gone "zombie" on me: Using netstat, I can see my connections appear on the server with state SYN_RECV: tcp 0 0 0.0.0.0:10003 0.0.0.0:* LISTEN 1001 175227186 24400/java tcp 0 0 (server ip):10003 (client ip):57533 SYN_RECV 1001 0 - tcp 378 0 (server ip):10003 (client ip):55296 CLOSE_WAIT 0 0 - tcp 378 0 (server ip):10003 (client ip):55297 CLOSE_WAIT 0 0 - tcp 378 0 (server ip):10003 (client ip):55304 CLOSE_WAIT 0 0 - tcp 378 0 (server ip):10003 (client ip):55320 CLOSE_WAIT 0 0 - tcp 378 0 (server ip):10003 (client ip):55321 CLOSE_WAIT 0 0 - tcp 378 0 (server ip):10003 (client ip):55716 CLOSE_WAIT 0 0 - tcp 378 0 (server ip):10003 (client ip):55713 CLOSE_WAIT 0 0 - tcp 378 0 (server ip):10003 (client ip):55715 CLOSE_WAIT 0 0 - tcp 378 0 (server ip):10003 (client ip):55725 CLOSE_WAIT 0 0 - tcp 378 0 (server ip):10003 (client ip):54936 CLOSE_WAIT 0 0 - Netstat on the client shows: tcp 0 0 (client ip):57533 (server ip):10003 ESTABLISHED 0 13975972 - (This specific connection was created using "nc server 10003") A full thread dump via "killall -3 java" produces the following: Full thread dump Java HotSpot(TM) Client VM (1.4.2-b28 mixed mode): "HttpProcessor[10003][888]" daemon prio=1 tid=0x4c206b08 nid=0x6503 in Object.wait() [3fdff000..3fdff908] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:429) at org.apache.catalina.connector.http.HttpProcessor.await(HttpProcessor.java:391) - locked <0x45bcaf10> (a org.apache.catalina.connector.http.HttpProcessor) at org.apache.catalina.connector.http.HttpProcessor.run(HttpProcessor.java:1145) at java.lang.Thread.run(Thread.java:534) "HttpProcessor[10003][887]" daemon prio=1 tid=0x4c206008 nid=0x6502 in Object.wait() [3ffff000..3ffff908] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:429) at org.apache.catalina.connector.http.HttpProcessor.await(HttpProcessor.java:391) - locked <0x45bcb240> (a org.apache.catalina.connector.http.HttpProcessor) at org.apache.catalina.connector.http.HttpProcessor.run(HttpProcessor.java:1145) at java.lang.Thread.run(Thread.java:534) (... hundreds more HttpProcessors with similar stack traces ...) "HttpProcessor[10003][1]" daemon prio=1 tid=0x081c3c40 nid=0x5f83 in Object.wait() [babff000..babff908] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:429) at org.apache.catalina.connector.http.HttpProcessor.await(HttpProcessor.java:391) - locked <0x452093b0> (a org.apache.catalina.connector.http.HttpProcessor) at org.apache.catalina.connector.http.HttpProcessor.run(HttpProcessor.java:1145) at java.lang.Thread.run(Thread.java:534) "HttpProcessor[10003][0]" daemon prio=1 tid=0x081e23f8 nid=0x5f82 in Object.wait() [badff000..badff908] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:429) at org.apache.catalina.connector.http.HttpProcessor.await(HttpProcessor.java:391) - locked <0x45209450> (a org.apache.catalina.connector.http.HttpProcessor) at org.apache.catalina.connector.http.HttpProcessor.run(HttpProcessor.java:1145) at java.lang.Thread.run(Thread.java:534) "HostConfig[localhost]" daemon prio=1 tid=0x081e1740 nid=0x5f80 waiting on condition [bb1ff000..bb1ff908] at java.lang.Thread.sleep(Native Method) at org.apache.catalina.startup.HostConfig.threadSleep(HostConfig.java:824) at org.apache.catalina.startup.HostConfig.run(HostConfig.java:848) at java.lang.Thread.run(Thread.java:534) "StandardManager[/tomcat-docs]" daemon prio=1 tid=0x081e1550 nid=0x5f7f waiting on condition [bb3ff000..bb3ff908] at java.lang.Thread.sleep(Native Method) at org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java:810) at org.apache.catalina.session.StandardManager.run(StandardManager.java:869) at java.lang.Thread.run(Thread.java:534) (.. several threads specific to our webapp which all are sleeping ..) "StandardManager[/server]" daemon prio=1 tid=0x4bf004c8 nid=0x5f5e waiting on condition [be5ff000..be5ff908] at java.lang.Thread.sleep(Native Method) at org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java:810) at org.apache.catalina.session.StandardManager.run(StandardManager.java:869) at java.lang.Thread.run(Thread.java:534) "StandardManager[/admin]" daemon prio=1 tid=0x08136d68 nid=0x5f5d waiting on condition [be7ff000..be7ff908] at java.lang.Thread.sleep(Native Method) at org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java:810) at org.apache.catalina.session.StandardManager.run(StandardManager.java:869) at java.lang.Thread.run(Thread.java:534) "StandardManager[/manager]" daemon prio=1 tid=0x084cc8e8 nid=0x5f5c waiting on condition [be9ff000..be9ff908] at java.lang.Thread.sleep(Native Method) at org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java:810) at org.apache.catalina.session.StandardManager.run(StandardManager.java:869) at java.lang.Thread.run(Thread.java:534) "Signal Dispatcher" daemon prio=1 tid=0x080a2888 nid=0x5f58 waiting on condition [0..0] "Finalizer" daemon prio=1 tid=0x0808af70 nid=0x5f56 in Object.wait() [bf3ff000..bf3ff908] at java.lang.Object.wait(Native Method) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111) - locked <0x449f31b8> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) "Reference Handler" daemon prio=1 tid=0x08089458 nid=0x5f55 in Object.wait() [bf5ff000..bf5ff908] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:429) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:115) - locked <0x449f3220> (a java.lang.ref.Reference$Lock) "main" prio=1 tid=0x08052c68 nid=0x5f50 runnable [bfffc000..bfffd4ec] at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353) - locked <0x45209590> (a java.net.PlainSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:448) at java.net.ServerSocket.accept(ServerSocket.java:419) at org.apache.catalina.core.StandardServer.await(StandardServer.java:527) at org.apache.catalina.startup.Catalina.start(Catalina.java:521) at org.apache.catalina.startup.Catalina.execute(Catalina.java:400) at org.apache.catalina.startup.Catalina.process(Catalina.java:180) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:203) "VM Thread" prio=1 tid=0x08086210 nid=0x5f54 runnable "VM Periodic Task Thread" prio=1 tid=0x080a4fa8 nid=0x5f5a waiting on condition "Suspend Checker Thread" prio=1 tid=0x080a1e50 nid=0x5f57 runnable The VM is still running, since "asynchronous" threads performing session unbinding events and other tasks keep procuding logging statements. I haven't changed any settings in /proc/sys/net/ipv4, so "tcp_syncookies" is 0 (no syn cookie protection), and both "tcp_syn_retries" and "tcp_synack_retries" contain the default(?) value "5". This is also true for the linux client. I have tried using various client machines to connect, including both linux and windows, to no success. Could this be a Java or Linux bug? --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
