thanx for the update, i'm using SSLSocketConnector now instead of SSLSocketChannelConnector (also because of CPU issue), since then we dont have CPU issue anymore. also have of tons of EOF but it doesnt hurt much, as long as the system runs stable.
it was helpfull to know that 7.6.3 doent solve your issue, so i will not rush for that cheers On Fri, Apr 27, 2012 at 1:07 AM, Devon Lazarus <[email protected]>wrote: > Hi,**** > > ** ** > > I just wanted to provide an update on this for the benefit of the list.*** > * > > ** ** > > We went ahead and switched to the BlockingChannelConnector. Unfortunately, > not only did that not solve our CPU usage issues, but it introduced some > network communication issues. Notably we started seeing ton’s of these in > the log files:**** > > ** ** > > Caused by: org.eclipse.jetty.io.EofException**** > > ** ** > > I tried tuning the tcp driver but that didn’t help at all. Also of note: > we’re using IPv6. I did try to force the JVM to use IPv4 but that didn’t > have an impact either.**** > > ** ** > > There was also a suggestion from the list to try 7.6.3 and we did that > too. We didn’t see any improvement in CPU and we still had the EoF errors. > **** > > ** ** > > For now I’ve gone back to the SelectChannelConnector with 7.6.3. The EoF > errors have subsided but our CPU consumption is too high for our liking. > We’ve instrumented the code and are looking at some other options > internally to understand what the issues might be.**** > > > Thanks for the suggestions, and I’ll report back to the list if I ever do > find the solution.**** > > ** ** > > Cheers,**** > > ** ** > > // devon**** > > ** ** > > ** ** > > *From:* [email protected] [mailto: > [email protected]] *On Behalf Of *Neil Bliss > *Sent:* Wednesday, April 18, 2012 4:51 PM > *To:* JETTY user mailing list > *Subject:* Re: [jetty-users] High CPU Load in Jetty 7.6.2 / JDK 1.7**** > > ** ** > > We had seen similar behavior when we used the SelectChannelConnector. In > our situation, we found that using the BlockingChannelConnector was a > better fit. That might be worth a try in your situation.**** > > On Wed, Apr 18, 2012 at 4:41 PM, Devon Lazarus <[email protected]> > wrote:**** > > Hi,**** > > **** > > I’ve been trying to get to the bottom of what I believe is a performance > problem in a Jetty application. Unfortunately, I’ve hit a road block and > could use some help. Googling hasn’t shed any light on the issue. Neither > has stackoverflow.**** > > **** > > Basically, I’m seeing very high CPU load without a lot of throughput. To > quantify that, we have a c1.medium (dual core) running Ubuntu 10.04LTS in > EC2 with Jetty 7.6.2 configured at 50 min/200 max threads, a single > acceptor, and the Oracle JDK 1.7_03 JVM configured with -mx2048m –ms512m, > and some GC tuning.**** > > **** > > We’re using a custom load generation tool to find the point at which the > server falls over. However, we’re hitting 90% CPU burn with loads well over > 2 while handling ~25 concurrent transactions/second (throughput was 63 tps > with ~400ms response times). That just doesn’t seem right (way too low).** > ** > > **** > > In profiling the JVM regardless of load (low or under siege), we can see > clearly that 99% of our CPU time is spent in one of two places:**** > > **** > > sun.nio.ch.SelectorImpl.select() at 66%**** > > sun.nio.ch.ServerSocketChannelImpl.accept() at 33%**** > > **** > > with all of that time being consumed by the acceptor/selector threads > dumped below in equal parts.**** > > **** > > In looking at the thread dump, we have many threads that look like this:** > ** > > **** > > "qtp665633643-88" - Thread t@88**** > > java.lang.Thread.State: TIMED_WAITING**** > > at sun.misc.Unsafe.park(Native Method)**** > > - parking to wait for <2ef1b67e> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)**** > > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)**** > > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > **** > > at > org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:337) > **** > > at > org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:517) > **** > > at > org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:39) > **** > > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:563) > **** > > at java.lang.Thread.run(Thread.java:722)**** > > **** > > Locked ownable synchronizers:**** > > - None**** > > **** > > And I see the acceptor/selectors like this:**** > > **** > > "qtp665633643-83 Selector0" - Thread t@83**** > > java.lang.Thread.State: RUNNABLE**** > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)** > ** > > at > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)**** > > at > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)**** > > at > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)**** > > - locked <7b18010e> (a sun.nio.ch.Util$2)**** > > - locked <23bb2f20> (a > java.util.Collections$UnmodifiableSet)**** > > - locked <7557d06a> (a sun.nio.ch.EPollSelectorImpl)**** > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)*** > * > > at > org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:564) > **** > > at > org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:285)** > ** > > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) > **** > > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) > **** > > at java.lang.Thread.run(Thread.java:722)**** > > **** > > Locked ownable synchronizers:**** > > - None**** > > **** > > "qtp1586177080-28 Acceptor0 [email protected]:80" - Thread > t@28**** > > java.lang.Thread.State: RUNNABLE**** > > at sun.nio.ch.ServerSocketChannelImpl.accept0(Native > Method)**** > > at > sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226) > **** > > - locked <58341b55> (a java.lang.Object)**** > > at > org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104) > **** > > at > org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933) > **** > > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) > **** > > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) > **** > > at java.lang.Thread.run(Thread.java:722)**** > > **** > > Locked ownable synchronizers:**** > > - None**** > > **** > > "qtp1586177080-27 Selector0" - Thread t@27**** > > java.lang.Thread.State: RUNNABLE**** > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)** > ** > > at > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)**** > > at > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)**** > > at > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)**** > > - locked <be7202b> (a sun.nio.ch.Util$2)**** > > - locked <2761d665> (a > java.util.Collections$UnmodifiableSet)**** > > - locked <4b49db96> (a sun.nio.ch.EPollSelectorImpl)**** > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)*** > * > > at > org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:564) > **** > > at > org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:285)** > ** > > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) > **** > > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) > **** > > at java.lang.Thread.run(Thread.java:722)**** > > **** > > Locked ownable synchronizers:**** > > - None**** > > **** > > Not a single thread in BLOCKED state.**** > > **** > > I know I only have a single acceptor configured when I have two > processors, but when I do configure two acceptors, I see the two acceptor > threads blocking each other (one RUNNABLE while the other BLOCKED and > switch back and forth after 10s of seconds).**** > > **** > > Any help is appreciated, and I apologize for the long email.**** > > **** > > Regards,**** > > **** > > Devon Lazarus, Internet Services Group**** > > Sonos, Inc.**** > > **** > > > _______________________________________________ > jetty-users mailing list > [email protected] > https://dev.eclipse.org/mailman/listinfo/jetty-users**** > > > > **** > > ** ** > > -- > Neil Bliss | Senior Engineer | 3Crowd Technologies, Inc. > c: +1.408.838.9632 | o: +1.415.445.4500**** > > _______________________________________________ > jetty-users mailing list > [email protected] > https://dev.eclipse.org/mailman/listinfo/jetty-users > >
_______________________________________________ jetty-users mailing list [email protected] https://dev.eclipse.org/mailman/listinfo/jetty-users
