Hi John, sounds like this bug in NIO:
http://jira.codehaus.org/browse/JETTY-937 http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6403933 I think recent versions of jetty work around this bug, or maybe try the non-NIO socket connector Kent On Tue, Dec 7, 2010 at 9:10 AM, John Russell <jjruss...@gmail.com> wrote: > Hi, > I'm using solr and have been load testing it for around 4 days. We use the > solrj client to communicate with a separate jetty based solr process on the > same box. > > After a few days solr's CPU% is now consistently at or above 100% (multiple > processors available) and the application using it is mostly not responding > because it times out talking to solr. I connected visual VM to the solr JVM > and found that out of the many btpool-# threads there are 4 that are pretty > much stuck in the running state 100% of the time. Their names are > > btpool0-1-Acceptor1 SelectChannelConnector @0.0.0.0:9983 > btpool0-2-Acceptor2 SelectChannelConnector @0.0.0.0:9983 > btpool0-3-Acceptor3 SelectChannelConnector @0.0.0.0:9983 > btpool0-9-Acceptor0 SelectChannelConnector @0.0.0.0:9983 > > > > The stacks are all the same > > "btpool0-2 - Acceptor2 SelectChannelConnector @ 0.0.0.0:9983" - Thread > t...@27 > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked <106a644> (a sun.nio.ch.Util$1) > - locked <18dd381> (a java.util.Collections$UnmodifiableSet) > - locked <38d07d> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at > org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:419) > at > org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:169) > at > org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) > at > org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:516) > at > org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442) > > Locked ownable synchronizers: > - None > > All of the other idle thread pool threads are just waiting for new tasks. > The active threads never seem to change, its always these 4. The selector > channel appears to be in the jetty code, receiving requests from our other > process through the solrj client. > > Does anyone know what this might mean or how to address it? Are these > running all the time because they are blocked on IO so not actually > consuming CPU? If so, what else might be? Is there a better way to figure > out what is pinning the CPU? > > Some more info that might be useful. > > 32 bit machine ( I know, I know) > 2.7GB of RAM for solr process ~2.5 is "used" > According to visual VM around 25% of CPU time is spent in GC with the rest > in application. > > Thanks for the help. > > John >
