Hi,
I'm upgrading from Tomcat 8.0.33.
I see that after a period of work requests remains "pending", for instance
I get all clients remaining
waiting for a response (parsing HTTP Response header) and no active thread
on my Tomcat.
This happens in my QA environment where I start several WebDriver/Unit
tests againts my web application. I have many cases, the most simple I this
is the following:
on the client side (a JAX-WS client):
"main" #1 prio=5 os_prio=0 tid=0x00007f7cf0009000 nid=0x7ddc runnable
[0x00007f7cf6e3f000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked <0x00000000fd159a18> (a java.io.BufferedInputStream)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1536)
- locked <0x00000000fcecee78> (a
sun.net.www.protocol.http.HttpURLConnection)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
- locked <0x00000000fcecee78> (a
sun.net.www.protocol.http.HttpURLConnection)
at
java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at
com.sun.xml.ws.transport.http.client.HttpClientTransport.readResponseCodeAndMessage(HttpClientTransport.java:209)
at
com.sun.xml.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:162)
at
com.sun.xml.ws.transport.http.client.HttpTransportPipe.processRequest(HttpTransportPipe.java:94)
at
com.sun.xml.ws.transport.DeferredTransportPipe.processRequest(DeferredTransportPipe.java:116)
at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:598)
at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:557)
at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:542)
at com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:439)
- locked <0x00000000fcbb7720> (a com.sun.xml.ws.api.pipe.Fiber)
at
com.sun.xml.ws.api.pipe.helper.AbstractTubeImpl.process(AbstractTubeImpl.java:112)
at com.sun.xml.xwss.XWSSClientPipe.process(XWSSClientPipe.java:160)
at
com.sun.xml.ws.api.pipe.helper.PipeAdapter.processRequest(PipeAdapter.java:115)
at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:598)
at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:557)
at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:542)
at com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:439)
- locked <0x00000000fcbb7720> (a com.sun.xml.ws.api.pipe.Fiber)
at com.sun.xml.ws.client.Stub.process(Stub.java:222)
at com.sun.xml.ws.client.sei.SEIStub.doProcess(SEIStub.java:135)
at
com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:109)
at
com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:89)
at com.sun.xml.ws.client.sei.SEIStub.invoke(SEIStub.java:118)
in the server side:
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.92-b14 mixed mode):
"anInnocuousThread" #861 daemon prio=5 os_prio=0 tid=0x00007ff23c0de800
nid=0xdb26 runnable [0x00007ff1ad379000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPoll.epollWait(Native Method)
at sun.nio.ch.EPollPort$EventHandlerTask.poll(EPollPort.java:194)
at sun.nio.ch.EPollPort$EventHandlerTask.run(EPollPort.java:268)
at java.lang.Thread.run(Thread.java:745)
at sun.misc.InnocuousThread.run(InnocuousThread.java:74)
(lots of this kind....) (HTTPS Connector I think)
"https-jsse-nio2-10.168.10.55-8443-exec-130" #572 daemon prio=5 os_prio=0
tid=0x00007ff23c0a0000 nid=0xd4a2 waiting on condition [0x00007ff1b2bbf000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000080f1d420> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
(lots of this kind....) (HTTP Connector I think)
"http-nio2-10.168.10.55-8080-exec-21" #281 daemon prio=5 os_prio=0
tid=0x00007ff2340e7000 nid=0xd262 waiting on condition [0x00007ff1af995000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000080f0cdb0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
"http-nio2-10.168.10.55-18443-Acceptor-0" #89 daemon prio=5 os_prio=0
tid=0x00007ff2581f2000 nid=0xd0dc waiting on condition [0x00007ff1b6eed000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000084c82768> (a
java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
at sun.nio.ch.PendingFuture.get(PendingFuture.java:180)
at
org.apache.tomcat.util.net.Nio2Endpoint$Acceptor.run(Nio2Endpoint.java:433)
at java.lang.Thread.run(Thread.java:745)
"https-jsse-nio2-10.168.10.55-8443-AsyncTimeout" #88 daemon prio=5
os_prio=0 tid=0x00007ff2581f1800 nid=0xd0db waiting on condition
[0x00007ff1b6fee000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.coyote.AbstractProtocol$AsyncTimeout.run(AbstractProtocol.java:1124)
at java.lang.Thread.run(Thread.java:745)
"https-jsse-nio2-10.168.10.55-8443-Acceptor-0" #87 daemon prio=5 os_prio=0
tid=0x00007ff2581f0800 nid=0xd0da waiting on condition [0x00007ff1b70ef000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000b6fbd830> (a
java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
at sun.nio.ch.PendingFuture.get(PendingFuture.java:180)
at
org.apache.tomcat.util.net.Nio2Endpoint$Acceptor.run(Nio2Endpoint.java:433)
at java.lang.Thread.run(Thread.java:745)
"http-nio2-10.168.10.55-8080-AsyncTimeout" #86 daemon prio=5 os_prio=0
tid=0x00007ff2581ef800 nid=0xd0d9 waiting on condition [0x00007ff1b71f0000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.coyote.AbstractProtocol$Asyn
"http-nio2-10.168.10.55-8080-Acceptor-0" #85 daemon prio=5 os_prio=0
tid=0x00007ff2581ee800 nid=0xd0d8 waiting on condition [0x00007ff1b72f1000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000b3c08980> (a
java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
at sun.nio.ch.PendingFuture.get(PendingFuture.java:180)
at
org.apache.tomcat.util.net.Nio2Endpoint$Acceptor.run(Nio2Endpoint.java:433)
at java.lang.Thread.run(Thread.java:745)
"ContainerBackgroundProcessor[StandardEngine[Tomcat]]" #83 daemon prio=5
os_prio=0 tid=0x00007ff2581ec800 nid=0xd0d6 waiting on condition
[0x00007ff1b74f3000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1355)
at java.lang.Thread.run(Thread.java:745)
"anInnocuousThread" #35 daemon prio=5 os_prio=0 tid=0x00007ff25909c800
nid=0xd099 runnable [0x00007ff1fc8bd000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPoll.epollWait(Native Method)
at sun.nio.ch.EPollPort$EventHandlerTask.poll(EPollPort.java:194)
at sun.nio.ch.EPollPort$EventHandlerTask.run(EPollPort.java:268)
at java.lang.Thread.run(Thread.java:745)
at sun.misc.InnocuousThread.run(InnocuousThread.java:74)
Thanks
Enrico Olivelli
