we have the same issue with NIO & had to switch back to BlockingIO, (jdk 1.6.0.31, linux 3.1)
http://dev.eclipse.org/mhonarc/lists/jetty-users/msg02163.html On Thu, Apr 19, 2012 at 1:51 AM, Neil Bliss <[email protected]> wrote: > We had seen similar behavior when we used the SelectChannelConnector. In > our situation, we found that using the BlockingChannelConnector was a > better fit. That might be worth a try in your situation. > > On Wed, Apr 18, 2012 at 4:41 PM, Devon Lazarus <[email protected]>wrote: > >> Hi,**** >> >> ** ** >> >> I’ve been trying to get to the bottom of what I believe is a performance >> problem in a Jetty application. Unfortunately, I’ve hit a road block and >> could use some help. Googling hasn’t shed any light on the issue. Neither >> has stackoverflow.**** >> >> ** ** >> >> Basically, I’m seeing very high CPU load without a lot of throughput. To >> quantify that, we have a c1.medium (dual core) running Ubuntu 10.04LTS in >> EC2 with Jetty 7.6.2 configured at 50 min/200 max threads, a single >> acceptor, and the Oracle JDK 1.7_03 JVM configured with -mx2048m –ms512m, >> and some GC tuning.**** >> >> ** ** >> >> We’re using a custom load generation tool to find the point at which the >> server falls over. However, we’re hitting 90% CPU burn with loads well over >> 2 while handling ~25 concurrent transactions/second (throughput was 63 tps >> with ~400ms response times). That just doesn’t seem right (way too low).* >> *** >> >> ** ** >> >> In profiling the JVM regardless of load (low or under siege), we can see >> clearly that 99% of our CPU time is spent in one of two places:**** >> >> ** ** >> >> sun.nio.ch.SelectorImpl.select() at 66%**** >> >> sun.nio.ch.ServerSocketChannelImpl.accept() at 33%**** >> >> ** ** >> >> with all of that time being consumed by the acceptor/selector threads >> dumped below in equal parts.**** >> >> ** ** >> >> In looking at the thread dump, we have many threads that look like this:* >> *** >> >> ** ** >> >> "qtp665633643-88" - Thread t@88**** >> >> java.lang.Thread.State: TIMED_WAITING**** >> >> at sun.misc.Unsafe.park(Native Method)**** >> >> - parking to wait for <2ef1b67e> (a >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)*** >> * >> >> at >> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)*** >> * >> >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) >> **** >> >> at >> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:337) >> **** >> >> at >> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:517) >> **** >> >> at >> org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:39) >> **** >> >> at >> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:563) >> **** >> >> at java.lang.Thread.run(Thread.java:722)**** >> >> ** ** >> >> Locked ownable synchronizers:**** >> >> - None**** >> >> ** ** >> >> And I see the acceptor/selectors like this:**** >> >> ** ** >> >> "qtp665633643-83 Selector0" - Thread t@83**** >> >> java.lang.Thread.State: RUNNABLE**** >> >> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)* >> *** >> >> at >> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)**** >> >> at >> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)**** >> >> at >> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)**** >> >> - locked <7b18010e> (a sun.nio.ch.Util$2)**** >> >> - locked <23bb2f20> (a >> java.util.Collections$UnmodifiableSet)**** >> >> - locked <7557d06a> (a sun.nio.ch.EPollSelectorImpl)**** >> >> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)** >> ** >> >> at >> org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:564) >> **** >> >> at >> org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:285)* >> *** >> >> at >> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) >> **** >> >> at >> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) >> **** >> >> at java.lang.Thread.run(Thread.java:722)**** >> >> ** ** >> >> Locked ownable synchronizers:**** >> >> - None**** >> >> ** ** >> >> "qtp1586177080-28 Acceptor0 [email protected]:80" - Thread >> t@28**** >> >> java.lang.Thread.State: RUNNABLE**** >> >> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native >> Method)**** >> >> at >> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226) >> **** >> >> - locked <58341b55> (a java.lang.Object)**** >> >> at >> org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104) >> **** >> >> at >> org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933) >> **** >> >> at >> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) >> **** >> >> at >> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) >> **** >> >> at java.lang.Thread.run(Thread.java:722)**** >> >> ** ** >> >> Locked ownable synchronizers:**** >> >> - None**** >> >> ** ** >> >> "qtp1586177080-27 Selector0" - Thread t@27**** >> >> java.lang.Thread.State: RUNNABLE**** >> >> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)* >> *** >> >> at >> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)**** >> >> at >> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)**** >> >> at >> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)**** >> >> - locked <be7202b> (a sun.nio.ch.Util$2)**** >> >> - locked <2761d665> (a >> java.util.Collections$UnmodifiableSet)**** >> >> - locked <4b49db96> (a sun.nio.ch.EPollSelectorImpl)**** >> >> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)** >> ** >> >> at >> org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:564) >> **** >> >> at >> org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:285)* >> *** >> >> at >> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) >> **** >> >> at >> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) >> **** >> >> at java.lang.Thread.run(Thread.java:722)**** >> >> ** ** >> >> Locked ownable synchronizers:**** >> >> - None**** >> >> ** ** >> >> Not a single thread in BLOCKED state.**** >> >> ** ** >> >> I know I only have a single acceptor configured when I have two >> processors, but when I do configure two acceptors, I see the two acceptor >> threads blocking each other (one RUNNABLE while the other BLOCKED and >> switch back and forth after 10s of seconds).**** >> >> ** ** >> >> Any help is appreciated, and I apologize for the long email.**** >> >> ** ** >> >> Regards,**** >> >> ** ** >> >> Devon Lazarus, Internet Services Group**** >> >> Sonos, Inc.**** >> >> ** ** >> >> _______________________________________________ >> jetty-users mailing list >> [email protected] >> https://dev.eclipse.org/mailman/listinfo/jetty-users >> >> > > > -- > Neil Bliss | Senior Engineer | 3Crowd Technologies, Inc. > c: +1.408.838.9632 | o: +1.415.445.4500 > > _______________________________________________ > jetty-users mailing list > [email protected] > https://dev.eclipse.org/mailman/listinfo/jetty-users > >
_______________________________________________ jetty-users mailing list [email protected] https://dev.eclipse.org/mailman/listinfo/jetty-users
