Hi,
I've been trying to get to the bottom of what I believe is a performance
problem in a Jetty application. Unfortunately, I've hit a road block and
could use some help. Googling hasn't shed any light on the issue.
Neither has stackoverflow.
Basically, I'm seeing very high CPU load without a lot of throughput. To
quantify that, we have a c1.medium (dual core) running Ubuntu 10.04LTS
in EC2 with Jetty 7.6.2 configured at 50 min/200 max threads, a single
acceptor, and the Oracle JDK 1.7_03 JVM configured with -mx2048m
-ms512m, and some GC tuning.
We're using a custom load generation tool to find the point at which the
server falls over. However, we're hitting 90% CPU burn with loads well
over 2 while handling ~25 concurrent transactions/second (throughput was
63 tps with ~400ms response times). That just doesn't seem right (way
too low).
In profiling the JVM regardless of load (low or under siege), we can see
clearly that 99% of our CPU time is spent in one of two places:
sun.nio.ch.SelectorImpl.select() at 66%
sun.nio.ch.ServerSocketChannelImpl.accept() at 33%
with all of that time being consumed by the acceptor/selector threads
dumped below in equal parts.
In looking at the thread dump, we have many threads that look like this:
"qtp665633643-88" - Thread t@88
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <2ef1b67e> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
aitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:3
37)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadP
ool.java:517)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPo
ol.java:39)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.ja
va:563)
at java.lang.Thread.run(Thread.java:722)
Locked ownable synchronizers:
- None
And I see the acceptor/selectors like this:
"qtp665633643-83 Selector0" - Thread t@83
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <7b18010e> (a sun.nio.ch.Util$2)
- locked <23bb2f20> (a
java.util.Collections$UnmodifiableSet)
- locked <7557d06a> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorMana
ger.java:564)
at
org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:285)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.j
ava:599)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.ja
va:534)
at java.lang.Thread.run(Thread.java:722)
Locked ownable synchronizers:
- None
"qtp1586177080-28 Acceptor0 [email protected]:80" - Thread
t@28
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native
Method)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:2
26)
- locked <58341b55> (a java.lang.Object)
at
org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannel
Connector.java:104)
at
org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnecto
r.java:933)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.j
ava:599)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.ja
va:534)
at java.lang.Thread.run(Thread.java:722)
Locked ownable synchronizers:
- None
"qtp1586177080-27 Selector0" - Thread t@27
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <be7202b> (a sun.nio.ch.Util$2)
- locked <2761d665> (a
java.util.Collections$UnmodifiableSet)
- locked <4b49db96> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorMana
ger.java:564)
at
org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:285)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.j
ava:599)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.ja
va:534)
at java.lang.Thread.run(Thread.java:722)
Locked ownable synchronizers:
- None
Not a single thread in BLOCKED state.
I know I only have a single acceptor configured when I have two
processors, but when I do configure two acceptors, I see the two
acceptor threads blocking each other (one RUNNABLE while the other
BLOCKED and switch back and forth after 10s of seconds).
Any help is appreciated, and I apologize for the long email.
Regards,
Devon Lazarus, Internet Services Group
Sonos, Inc.
_______________________________________________
jetty-users mailing list
[email protected]
https://dev.eclipse.org/mailman/listinfo/jetty-users
