solr 5.5.2 dump threads - threads blocked in org.eclipse.jetty.util.BlockingArrayQueue

[elisabeth benoit]

Hello,


We are perf testing solr 5.5.2 (with a limit test, i.e. sending as much
queries/sec as possible) and we see the cpu never goes over 20%, and
threads are blocked in org.eclipse.jetty.util.BlockingArrayQueue, as we can
see in solr admin interface thread dumps

qtp706277948-757 (757)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2c4a56cb

   - sun.misc.Unsafe.park​(Native Method)
   - java.util.concurrent.locks.LockSupport.parkNanos​(LockSupport.java:215)
   -
   
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos​(AbstractQueuedSynchronizer.java:2078)
   -
   org.eclipse.jetty.util.BlockingArrayQueue.poll​(BlockingArrayQueue.java:389)
   -
   
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll​(QueuedThreadPool.java:531)
   -
   
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700​(QueuedThreadPool.java:47)
   -
   
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run​(QueuedThreadPool.java:590)
   - java.lang.Thread.run​(Thread.java:745)


We changed two things in jetty configuration,

maxThreads value in /opt/solr/server/solr/jetty.xml

<Set name="maxThreads" type="int"><Property name="solr.jetty.threads.max"
default="400"/></Set>


and we activated the request log, i.e. uncommented the lines



    <Ref id="Handlers">

      <Call name="addHandler">

        <Arg>

          <New id="RequestLog"
class="org.eclipse.jetty.server.handler.RequestLogHandler">

            <Set name="requestLog">

              <New id="RequestLogImpl"
class="org.eclipse.jetty.server.NCSARequestLog">

                <Set name="filename">

                   /var/solr/logs/requests.log

                </Set>

                <Set name="filenameDateFormat">yyyy_MM_dd</Set>

                <Set name="retainDays">90</Set>

                <Set name="append">true</Set>

                <Set name="extended">false</Set>

                <Set name="logCookies">false</Set>

                <Set name="LogTimeZone">UTC</Set>

                <Set name="logLatency">true</Set>

              </New>

            </Set>

          </New>

        </Arg>

      </Call>

    </Ref>


in jetty.xml


We had the same result with maxThreads=10000 (default value in solr
install).


Did anyone experiment the same issue with solr 5?


Best regards,

Elisabeth