Guys, thanks for all the suggestions I will be trying them, one at a time. Imho it's too early to give up and look for another tool, I'll try to work on configuration and see what happens. The NRT looks quite promising, there are also tons of config options to change. As for now, I have made the updates less frequent - about once every 30 seconds (but now the batches are bigger, about 150-200 documents per update). I'll see if this makes SOLR more stable or users more aggressive. Unfortunately I have no resources for experimenting so I'll keep making small changes to production system and observing the effects. Shawn, I have given the JVM about 2 GB of memory but it's only using 300 MB so I don't think there's memory shortage now. The whole index is about 2 GB in size but I think there aren't enough queries to fill up the cache and make SOLR load everything in memory.
Below i'm pasting the thread dump taken when the update was hung (it's also attached to the first message of this topic) Best regards, RG <solr> <core>example</core> <system> <jvm> <version>20.5-b03</version> <name>Java HotSpot(TM) 64-Bit Server VM</name> </jvm> <threadCount> <current>31</current> <peak>32</peak> <daemon>8</daemon> </threadCount> <threadDump> <thread> <id>39</id> <name>pool-4-thread-1</name> <state>WAITING</state> <lock>java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@765bc9b8 </lock> <cpuTime>312,5000ms</cpuTime> <userTime>265,6250ms</userTime> <stackTrace> <line>at sun.misc.Unsafe.park(Native Method) </line> <line>at java.util.concurrent.locks.LockSupport.park(Unknown Source) </line> <line>at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source) </line> <line>at java.util.concurrent.DelayQueue.take(Unknown Source) </line> <line>at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) </line> <line>at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) </line> <line>at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) </line> <line>at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) </line> <line>at java.lang.Thread.run(Unknown Source) </line> </stackTrace> </thread> <thread> <id>38</id> <name>pool-2-thread-1</name> <state>WAITING</state> <lock>java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4188bbd </lock> <cpuTime>6484,3750ms</cpuTime> <userTime>5546,8750ms</userTime> <stackTrace> <line>at sun.misc.Unsafe.park(Native Method) </line> <line>at java.util.concurrent.locks.LockSupport.park(Unknown Source) </line> <line>at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source) </line> <line>at java.util.concurrent.DelayQueue.take(Unknown Source) </line> <line>at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) </line> <line>at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) </line> <line>at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) </line> <line>at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) </line> <line>at java.lang.Thread.run(Unknown Source) </line> </stackTrace> </thread> <thread> <id>37</id> <name>DestroyJavaVM</name> <state>RUNNABLE</state> <cpuTime>4906,2500ms</cpuTime> <userTime>4484,3750ms</userTime> <stackTrace> </stackTrace> </thread> <thread> <id>36</id> <name>qtp1033068770-36</name> <state>TIMED_WAITING</state> <lock>java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@677e2764 </lock> <cpuTime>134968,7500ms</cpuTime> <userTime>114984,3750ms</userTime> <stackTrace> <line>at sun.misc.Unsafe.park(Native Method) </line> <line>at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) </line> <line>at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source) </line> <line>at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320) </line> <line>at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:480) </line> <line>at java.lang.Thread.run(Unknown Source) </line> </stackTrace> </thread> <thread> <id>35</id> <name>qtp1033068770-35</name> <state>RUNNABLE</state> <cpuTime>147390,6250ms</cpuTime> <userTime>126593,7500ms</userTime> <stackTrace> <line>at sun.management.ThreadImpl.getThreadInfo1(Native Method) </line> <line>at sun.management.ThreadImpl.getThreadInfo(Unknown Source) </line> <line>at org.apache.jsp.admin.threaddump_jsp._jspService(org.apache.jsp.admin.threaddump_jsp:264) </line> <line>at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:109) </line> <line>at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) </line> <line>at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:389) </line> <line>at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:486) </line> <line>at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:380) </line> <line>at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) </line> <line>at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:534) </line> <line>at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:475) </line> <line>at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119) </line> <line>at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:516) </line> <line>at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) </line> <line>at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:929) </line> <line>at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:403) </line> <line>at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:184) </line> <line>at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:864) </line> <line>at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) </line> <line>at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:285) </line> <line>at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:116) </line> <line>at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:275) </line> <line>at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1322) </line> <line>at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:473) </line> <line>at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119) </line> <line>at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:479) </line> <line>at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) </line> <line>at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:929) </line> <line>at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:403) </line> <line>at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:184) </line> <line>at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:864) </line> <line>at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) </line> <line>at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:247) </line> <line>at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:151) </line> <line>at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:114) </line> <line>at org.eclipse.jetty.server.Server.handle(Server.java:352) </line> <line>at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:596) </line> <line>at org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:1051) </line> <line>at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:590) </line> <line>at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:212) </line> <line>at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:426) </line> <line>at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:508) </line> <line>at org.eclipse.jetty.io.nio.SelectChannelEndPoint.access$000(SelectChannelEndPoint.java:34) </line> <line>at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:40) </line> <line>at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:451) </line> <line>at java.lang.Thread.run(Unknown Source) </line> </stackTrace> </thread> <thread> <id>34</id> <name>qtp1033068770-34</name> <state>TIMED_WAITING</state> <lock>java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@677e2764 </lock> <cpuTime>131859,3750ms</cpuTime> <userTime>112515,6250ms</userTime> <stackTrace> <line>at sun.misc.Unsafe.park(Native Method) </line> <line>at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) </line> <line>at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source) </line> <line>at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320) </line> <line>at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:480) </line> <line>at java.lang.Thread.run(Unknown Source) </line> </stackTrace> </thread> <thread> <id>32</id> <name>qtp1033068770-32</name> <state>TIMED_WAITING</state> <lock>java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@677e2764 </lock> <cpuTime>143156,2500ms</cpuTime> <userTime>121687,5000ms</userTime> <stackTrace> <line>at sun.misc.Unsafe.park(Native Method) </line> <line>at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) </line> <line>at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source) </line> <line>at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320) </line> <line>at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:480) </line> <line>at java.lang.Thread.run(Unknown Source) </line> </stackTrace> </thread> <thread> <id>31</id> <name>qtp1033068770-31</name> <state>RUNNABLE</state> <inNative/> <cpuTime>703,1250ms</cpuTime> <userTime>390,6250ms</userTime> <stackTrace> <line>at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) </line> <line>at sun.nio.ch.ServerSocketChannelImpl.accept(Unknown Source) </line> <line>at org.eclipse.jetty.server.nio.SelectChannelConnector$2.run(SelectChannelConnector.java:287) </line> <line>at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:451) </line> <line>at java.lang.Thread.run(Unknown Source) </line> </stackTrace> </thread> <thread> <id>30</id> <name>qtp1033068770-30</name> <state>TIMED_WAITING</state> <lock>java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@677e2764 </lock> <cpuTime>129718,7500ms</cpuTime> <userTime>110812,5000ms</userTime> <stackTrace> <line>at sun.misc.Unsafe.park(Native Method) </line> <line>at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) </line> <line>at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source) </line> <line>at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320) </line> <line>at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:480) </line> <line>at java.lang.Thread.run(Unknown Source) </line> </stackTrace> </thread> <thread> <id>29</id> <name>qtp1033068770-29 - Acceptor1 SelectChannelConnector@0.0.0.0:9070 </name> <state>RUNNABLE</state> <inNative/> <cpuTime>5734,3750ms</cpuTime> <userTime>2875,0000ms</userTime> <stackTrace> <line>at sun.nio.ch.WindowsSelectorImpl.discardUrgentData(Native Method) </line> <line>at sun.nio.ch.WindowsSelectorImpl.access$2200(Unknown Source) </line> <line>at sun.nio.ch.WindowsSelectorImpl$SubSelector.processFDSet(Unknown Source) </line> <line>at sun.nio.ch.WindowsSelectorImpl$SubSelector.processSelectedKeys(Unknown Source) </line> <line>at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$2900(Unknown Source) </line> <line>at sun.nio.ch.WindowsSelectorImpl.updateSelectedKeys(Unknown Source) </line> <line>at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source) </line> <line>at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source) </line> <line>at sun.nio.ch.SelectorImpl.select(Unknown Source) </line> <line>at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:481) </line> <line>at org.eclipse.jetty.io.nio.SelectorManager.doSelect(SelectorManager.java:219) </line> <line>at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:122) </line> <line>at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:837) </line> <line>at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:451) </line> <line>at java.lang.Thread.run(Unknown Source) </line> </stackTrace> </thread> <thread> <id>28</id> <name>qtp1033068770-28</name> <state>TIMED_WAITING</state> <lock>java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@677e2764 </lock> <cpuTime>137921,8750ms</cpuTime> <userTime>118078,1250ms</userTime> <stackTrace> <line>at sun.misc.Unsafe.park(Native Method) </line> <line>at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) </line> <line>at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source) </line> <line>at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320) </line> <line>at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:480) </line> <line>at java.lang.Thread.run(Unknown Source) </line> </stackTrace> </thread> <thread> <id>27</id> <name>qtp1033068770-27 - Acceptor0 SelectChannelConnector@0.0.0.0:9070 </name> <state>RUNNABLE</state> <inNative/> <cpuTime>6859,3750ms</cpuTime> <userTime>3890,6250ms</userTime> <stackTrace> <line>at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) </line> <line>at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source) </line> <line>at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source) </line> <line>at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source) </line> <line>at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source) </line> <line>at sun.nio.ch.SelectorImpl.select(Unknown Source) </line> <line>at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:481) </line> <line>at org.eclipse.jetty.io.nio.SelectorManager.doSelect(SelectorManager.java:219) </line> <line>at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:122) </line> <line>at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:837) </line> <line>at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:451) </line> <line>at java.lang.Thread.run(Unknown Source) </line> </stackTrace> </thread> <thread> <id>26</id> <name>qtp1033068770-26</name> <state>TIMED_WAITING</state> <lock>java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@677e2764 </lock> <cpuTime>132546,8750ms</cpuTime> <userTime>112984,3750ms</userTime> <stackTrace> <line>at sun.misc.Unsafe.park(Native Method) </line> <line>at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) </line> <line>at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source) </line> <line>at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320) </line> <line>at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:480) </line> <line>at java.lang.Thread.run(Unknown Source) </line> </stackTrace> </thread> <thread> <id>25</id> <name>Scanner-1</name> <state>TIMED_WAITING</state> <lock>java.util.TaskQueue@b4c4b51</lock> <cpuTime>46,8750ms</cpuTime> <userTime>46,8750ms</userTime> <stackTrace> <line>at java.lang.Object.wait(Native Method) </line> <line>at java.util.TimerThread.mainLoop(Unknown Source) </line> <line>at java.util.TimerThread.run(Unknown Source) </line> </stackTrace> </thread> <thread> <id>24</id> <name>TransparentProxy-24</name> <state>TIMED_WAITING</state> <lock>java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3de4b8a2 </lock> <cpuTime>0,0000ms</cpuTime> <userTime>0,0000ms</userTime> <stackTrace> <line>at sun.misc.Unsafe.park(Native Method) </line> <line>at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) </line> <line>at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source) </line> <line>at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320) </line> <line>at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:480) </line> <line>at java.lang.Thread.run(Unknown Source) </line> </stackTrace> </thread> <thread> <id>23</id> <name>TransparentProxy-23</name> <state>TIMED_WAITING</state> <lock>java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3de4b8a2 </lock> <cpuTime>0,0000ms</cpuTime> <userTime>0,0000ms</userTime> <stackTrace> <line>at sun.misc.Unsafe.park(Native Method) </line> <line>at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) </line> <line>at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source) </line> <line>at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320) </line> <line>at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:480) </line> <line>at java.lang.Thread.run(Unknown Source) </line> </stackTrace> </thread> <thread> <id>22</id> <name>TransparentProxy-22</name> <state>TIMED_WAITING</state> <lock>java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3de4b8a2 </lock> <cpuTime>0,0000ms</cpuTime> <userTime>0,0000ms</userTime> <stackTrace> <line>at sun.misc.Unsafe.park(Native Method) </line> <line>at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) </line> <line>at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source) </line> <line>at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320) </line> <line>at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:480) </line> <line>at java.lang.Thread.run(Unknown Source) </line> </stackTrace> </thread> <thread> <id>21</id> <name>TransparentProxy-21</name> <state>TIMED_WAITING</state> <lock>java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3de4b8a2 </lock> <cpuTime>15,6250ms</cpuTime> <userTime>15,6250ms</userTime> <stackTrace> <line>at sun.misc.Unsafe.park(Native Method) </line> <line>at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) </line> <line>at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source) </line> <line>at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320) </line> <line>at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:480) </line> <line>at java.lang.Thread.run(Unknown Source) </line> </stackTrace> </thread> <thread> <id>20</id> <name>TransparentProxy-20</name> <state>TIMED_WAITING</state> <lock>java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3de4b8a2 </lock> <cpuTime>0,0000ms</cpuTime> <userTime>0,0000ms</userTime> <stackTrace> <line>at sun.misc.Unsafe.park(Native Method) </line> <line>at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) </line> <line>at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source) </line> <line>at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320) </line> <line>at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:480) </line> <line>at java.lang.Thread.run(Unknown Source) </line> </stackTrace> </thread> <thread> <id>19</id> <name>TransparentProxy-19</name> <state>TIMED_WAITING</state> <lock>java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3de4b8a2 </lock> <cpuTime>0,0000ms</cpuTime> <userTime>0,0000ms</userTime> <stackTrace> <line>at sun.misc.Unsafe.park(Native Method) </line> <line>at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) </line> <line>at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source) </line> <line>at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320) </line> <line>at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:480) </line> <line>at java.lang.Thread.run(Unknown Source) </line> </stackTrace> </thread> <thread> <id>18</id> <name>TransparentProxy-18</name> <state>TIMED_WAITING</state> <cpuTime>15,6250ms</cpuTime> <userTime>15,6250ms</userTime> <stackTrace> <line>at java.lang.Thread.sleep(Native Method) </line> <line>at org.eclipse.jetty.client.HttpClient$1.run(HttpClient.java:483) </line> <line>at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:451) </line> <line>at java.lang.Thread.run(Unknown Source) </line> </stackTrace> </thread> <thread> <id>17</id> <name>TransparentProxy-17</name> <state>RUNNABLE</state> <inNative/> <cpuTime>171,8750ms</cpuTime> <userTime>140,6250ms</userTime> <stackTrace> <line>at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) </line> <line>at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source) </line> <line>at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source) </line> <line>at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source) </line> <line>at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source) </line> <line>at sun.nio.ch.SelectorImpl.select(Unknown Source) </line> <line>at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:481) </line> <line>at org.eclipse.jetty.io.nio.SelectorManager.doSelect(SelectorManager.java:219) </line> <line>at org.eclipse.jetty.client.SelectConnector.run(SelectConnector.java:163) </line> <line>at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:451) </line> <line>at java.lang.Thread.run(Unknown Source) </line> </stackTrace> </thread> <thread> <id>16</id> <name>HashSessionScavenger-1</name> <state>TIMED_WAITING</state> <lock>java.util.TaskQueue@7315b36c</lock> <cpuTime>0,0000ms</cpuTime> <userTime>0,0000ms</userTime> <stackTrace> <line>at java.lang.Object.wait(Native Method) </line> <line>at java.util.TimerThread.mainLoop(Unknown Source) </line> <line>at java.util.TimerThread.run(Unknown Source) </line> </stackTrace> </thread> <thread> <id>15</id> <name>Scanner-0</name> <state>TIMED_WAITING</state> <lock>java.util.TaskQueue@15b6b27b</lock> <cpuTime>93,7500ms</cpuTime> <userTime>15,6250ms</userTime> <stackTrace> <line>at java.lang.Object.wait(Native Method) </line> <line>at java.util.TimerThread.mainLoop(Unknown Source) </line> <line>at java.util.TimerThread.run(Unknown Source) </line> </stackTrace> </thread> <thread> <id>14</id> <name>pool-3-thread-1</name> <state>WAITING</state> <lock>java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@d7fb460 </lock> <cpuTime>1046,8750ms</cpuTime> <userTime>1000,0000ms</userTime> <stackTrace> <line>at sun.misc.Unsafe.park(Native Method) </line> <line>at java.util.concurrent.locks.LockSupport.park(Unknown Source) </line> <line>at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source) </line> <line>at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source) </line> <line>at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) </line> <line>at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) </line> <line>at java.lang.Thread.run(Unknown Source) </line> </stackTrace> </thread> <thread> <id>13</id> <name>pool-1-thread-1</name> <state>WAITING</state> <lock>java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b1b118c </lock> <cpuTime>4796,8750ms</cpuTime> <userTime>4578,1250ms</userTime> <stackTrace> <line>at sun.misc.Unsafe.park(Native Method) </line> <line>at java.util.concurrent.locks.LockSupport.park(Unknown Source) </line> <line>at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source) </line> <line>at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source) </line> <line>at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) </line> <line>at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) </line> <line>at java.lang.Thread.run(Unknown Source) </line> </stackTrace> </thread> <thread> <id>11</id> <name>HashSessionScavenger-0</name> <state>TIMED_WAITING</state> <lock>java.util.TaskQueue@6e1d1ca4</lock> <cpuTime>0,0000ms</cpuTime> <userTime>0,0000ms</userTime> <stackTrace> <line>at java.lang.Object.wait(Native Method) </line> <line>at java.util.TimerThread.mainLoop(Unknown Source) </line> <line>at java.util.TimerThread.run(Unknown Source) </line> </stackTrace> </thread> <thread> <id>5</id> <name>Attach Listener</name> <state>RUNNABLE</state> <cpuTime>0,0000ms</cpuTime> <userTime>0,0000ms</userTime> <stackTrace> </stackTrace> </thread> <thread> <id>4</id> <name>Signal Dispatcher</name> <state>RUNNABLE</state> <cpuTime>0,0000ms</cpuTime> <userTime>0,0000ms</userTime> <stackTrace> </stackTrace> </thread> <thread> <id>3</id> <name>Finalizer</name> <state>WAITING</state> <lock>java.lang.ref.ReferenceQueue$Lock@39438b76</lock> <cpuTime>78,1250ms</cpuTime> <userTime>62,5000ms</userTime> <stackTrace> <line>at java.lang.Object.wait(Native Method) </line> <line>at java.lang.ref.ReferenceQueue.remove(Unknown Source) </line> <line>at java.lang.ref.ReferenceQueue.remove(Unknown Source) </line> <line>at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source) </line> </stackTrace> </thread> <thread> <id>2</id> <name>Reference Handler</name> <state>WAITING</state> <lock>java.lang.ref.Reference$Lock@1790d59</lock> <cpuTime>78,1250ms</cpuTime> <userTime>62,5000ms</userTime> <stackTrace> <line>at java.lang.Object.wait(Native Method) </line> <line>at java.lang.Object.wait(Object.java:485) </line> <line>at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source) </line> </stackTrace> </thread> </threadDump> </system> </solr>