Hi,
I'm using solr and have been load testing it for around 4 days. We use the
solrj client to communicate with a separate jetty based solr process on the
same box.
After a few days solr's CPU% is now consistently at or above 100% (multiple
processors available) and the application using it is mostly not responding
because it times out talking to solr. I connected visual VM to the solr JVM
and found that out of the many btpool-# threads there are 4 that are pretty
much stuck in the running state 100% of the time. Their names are
btpool0-1-Acceptor1 SelectChannelConnector @0.0.0.0:9983
btpool0-2-Acceptor2 SelectChannelConnector @0.0.0.0:9983
btpool0-3-Acceptor3 SelectChannelConnector @0.0.0.0:9983
btpool0-9-Acceptor0 SelectChannelConnector @0.0.0.0:9983
The stacks are all the same
"btpool0-2 - Acceptor2 SelectChannelConnector @ 0.0.0.0:9983" - Thread
t...@27
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <106a644> (a sun.nio.ch.Util$1)
- locked <18dd381> (a java.util.Collections$UnmodifiableSet)
- locked <38d07d> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:419)
at
org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:169)
at
org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
at
org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:516)
at
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
Locked ownable synchronizers:
- None
All of the other idle thread pool threads are just waiting for new tasks.
The active threads never seem to change, its always these 4. The selector
channel appears to be in the jetty code, receiving requests from our other
process through the solrj client.
Does anyone know what this might mean or how to address it? Are these
running all the time because they are blocked on IO so not actually
consuming CPU? If so, what else might be? Is there a better way to figure
out what is pinning the CPU?
Some more info that might be useful.
32 bit machine ( I know, I know)
2.7GB of RAM for solr process ~2.5 is "used"
According to visual VM around 25% of CPU time is spent in GC with the rest
in application.
Thanks for the help.
John
