[
https://issues.apache.org/jira/browse/SOLR-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Thacker updated SOLR-6920:
--------------------------------
Attachment: SOLR-6920.patch
Few tests fail with this patch -
{code}
[junit4] Tests with failures:
[junit4] - org.apache.solr.cloud.BasicDistributedZk2Test.testDistribSearch
[junit4] - org.apache.solr.cloud.ShardSplitTest.testDistribSearch
[junit4] - org.apache.solr.cloud.SyncSliceTest.testDistribSearch
[junit4] - org.apache.solr.cloud.RecoveryZkTest.testDistribSearch
{code}
I picked ShardSplitTest.testDistribSearch and started investigating why is it
failing always with the patch -
I was seeing the following stack trace -
{code}
53340 T166 C65 P58357 oasc.SolrException.log ERROR SnapPull failed
:org.apache.solr.common.SolrException: Error opening new searcher
at
org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1603)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1715)
at
org.apache.solr.handler.SnapPuller.openNewSearcherAndUpdateCommitPoint(SnapPuller.java:680)
at
org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:496)
at
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:340)
at
org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:163)
at
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:447)
at
org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:235)
Caused by: java.io.FileNotFoundException: _1_2.liv in
dir=RAMDirectory@2fdfb9bd
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@91b3fd9
at
org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:655)
at
org.apache.lucene.store.Directory.openChecksumInput(Directory.java:110)
at
org.apache.lucene.codecs.lucene50.Lucene50LiveDocsFormat.readLiveDocs(Lucene50LiveDocsFormat.java:84)
at
org.apache.lucene.codecs.asserting.AssertingLiveDocsFormat.readLiveDocs(AssertingLiveDocsFormat.java:64)
at
org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:101)
at
org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:134)
at
org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:186)
at
org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:94)
at
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:430)
at
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:268)
at
org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:203)
at
org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1522)
... 7 more
53344 T166 C65 P58357 oasc.SolrException.log ERROR Error while trying to
recover:org.apache.solr.common.SolrException: Replication for recovery failed.
at
org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:166)
at
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:447)
at
org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:235)
{code}
This looked very similar to the problem we were trying to solve in SOLR-6640.
So I applied the patch from SOLR-6640 along with this patch and now I am
seeing these -
{code}
392008 T11 oasc.Diagnostics.logThreadDumps ERROR Gave up waiting for recovery
to finish. THREAD DUMP:
"qtp2004060070-174" Id=174 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42da41bf
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42da41bf
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"RecoveryThread-collection1_shard1_1_replica2" Id=168 TIMED_WAITING on
java.lang.Object@1398321c
at java.lang.Object.wait(Native Method)
- waiting on java.lang.Object@1398321c
at
org.apache.solr.update.DefaultSolrCoreState.closeIndexWriter(DefaultSolrCoreState.java:209)
- locked org.apache.solr.update.DefaultSolrCoreState@50d256b5
at
org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:469)
at
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:340)
at
org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:163)
at
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:447)
at
org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:235)
Number of locked synchronizers = 1
- java.util.concurrent.locks.ReentrantLock$NonfairSync@213aae87
"searcherExecutor-61-thread-1" Id=166 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6d01d650
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6d01d650
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"RecoveryThread-collection1_shard1_0_replica2" Id=163 TIMED_WAITING on
java.lang.Object@50e7ed31
at java.lang.Object.wait(Native Method)
- waiting on java.lang.Object@50e7ed31
at
org.apache.solr.update.DefaultSolrCoreState.closeIndexWriter(DefaultSolrCoreState.java:209)
- locked org.apache.solr.update.DefaultSolrCoreState@3e7f8d5e
at
org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:469)
at
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:340)
at
org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:163)
at
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:447)
at
org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:235)
Number of locked synchronizers = 1
- java.util.concurrent.locks.ReentrantLock$NonfairSync@281a3660
"searcherExecutor-56-thread-1" Id=161 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2a855576
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2a855576
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"searcherExecutor-52-thread-1" Id=156 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@17adcbc9
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@17adcbc9
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"searcherExecutor-48-thread-1" Id=154 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@131e0544
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@131e0544
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"OverseerThreadFactory-5-thread-4" Id=152 WAITING on
java.util.concurrent.SynchronousQueue$TransferStack@17ebbd2a
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.SynchronousQueue$TransferStack@17ebbd2a
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at
java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:925)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
...
"OverseerThreadFactory-5-thread-3" Id=146 WAITING on
java.util.concurrent.SynchronousQueue$TransferStack@17ebbd2a
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.SynchronousQueue$TransferStack@17ebbd2a
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at
java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:925)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
...
"OverseerThreadFactory-5-thread-2" Id=145 WAITING on
java.util.concurrent.SynchronousQueue$TransferStack@17ebbd2a
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.SynchronousQueue$TransferStack@17ebbd2a
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at
java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:925)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
...
"OverseerThreadFactory-5-thread-1" Id=144 WAITING on
java.util.concurrent.SynchronousQueue$TransferStack@17ebbd2a
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.SynchronousQueue$TransferStack@17ebbd2a
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at
java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:925)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
...
"coreZkRegister-39-thread-1" Id=136 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@75d01caa
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@75d01caa
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"Thread-57" Id=134 WAITING on java.lang.Object@320a1953
at java.lang.Object.wait(Native Method)
- waiting on java.lang.Object@320a1953
at java.lang.Object.wait(Object.java:503)
at org.apache.solr.core.CloserThread.run(CoreContainer.java:929)
"searcherExecutor-44-thread-1" Id=133 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3504b1cf
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3504b1cf
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"TEST-ShardSplitTest.testDistribSearch-seed#[8490788943A65A5D]-EventThread"
Id=129 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4735a83a
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4735a83a
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
"TEST-ShardSplitTest.testDistribSearch-seed#[8490788943A65A5D]-SendThread(127.0.0.1:57967)"
Id=128 RUNNABLE (in native)
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
at
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked sun.nio.ch.Util$2@7b7a6620
- locked java.util.Collections$UnmodifiableSet@685b5a62
- locked sun.nio.ch.KQueueSelectorImpl@2c205e57
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
"qtp860063164-124" Id=124 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@111b863
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@111b863
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp860063164-123" Id=123 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@111b863
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@111b863
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp860063164-122" Id=122 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@111b863
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@111b863
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp860063164-121" Id=121 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@111b863
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@111b863
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp860063164-120" Id=120 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@111b863
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@111b863
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp860063164-119" Id=119 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@111b863
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@111b863
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp860063164-118 Acceptor0 [email protected]:57992"
Id=118 RUNNABLE (in native)
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
- locked java.lang.Object@32606c21
at
org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
at
org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)
"qtp860063164-117 Selector0" Id=117 RUNNABLE (in native)
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
at
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked sun.nio.ch.Util$2@3f7ba2fc
- locked java.util.Collections$UnmodifiableSet@4d533b8b
- locked sun.nio.ch.KQueueSelectorImpl@27365e0f
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
at
org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
...
"HashSessionScavenger-4" Id=116 TIMED_WAITING on
java.util.TaskQueue@62593ee5
at java.lang.Object.wait(Native Method)
- waiting on java.util.TaskQueue@62593ee5
at java.util.TimerThread.mainLoop(Timer.java:552)
at java.util.TimerThread.run(Timer.java:505)
"coreZkRegister-30-thread-1" Id=113 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@51305ed4
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@51305ed4
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"Thread-45" Id=111 WAITING on java.lang.Object@320a1953
at java.lang.Object.wait(Native Method)
- waiting on java.lang.Object@320a1953
at java.lang.Object.wait(Object.java:503)
at org.apache.solr.core.CloserThread.run(CoreContainer.java:929)
"searcherExecutor-35-thread-1" Id=110 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3ea1d9f0
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3ea1d9f0
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"TEST-ShardSplitTest.testDistribSearch-seed#[8490788943A65A5D]-EventThread"
Id=107 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73eb3baf
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73eb3baf
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
"TEST-ShardSplitTest.testDistribSearch-seed#[8490788943A65A5D]-SendThread(127.0.0.1:57967)"
Id=106 RUNNABLE (in native)
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
at
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked sun.nio.ch.Util$2@4f823fdd
- locked java.util.Collections$UnmodifiableSet@19498b68
- locked sun.nio.ch.KQueueSelectorImpl@41550ef
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
"qtp695935058-102" Id=102 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@13747d01
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@13747d01
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp695935058-101" Id=101 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@13747d01
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@13747d01
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp695935058-100" Id=100 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@13747d01
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@13747d01
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp695935058-99" Id=99 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@13747d01
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@13747d01
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp695935058-98" Id=98 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@13747d01
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@13747d01
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp695935058-97" Id=97 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@13747d01
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@13747d01
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp695935058-96 Acceptor0 [email protected]:57988"
Id=96 RUNNABLE (in native)
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
- locked java.lang.Object@44bad6a1
at
org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
at
org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)
"qtp695935058-95 Selector0" Id=95 RUNNABLE (in native)
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
at
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked sun.nio.ch.Util$2@c24ea6a
- locked java.util.Collections$UnmodifiableSet@6211a266
- locked sun.nio.ch.KQueueSelectorImpl@7f8f54f6
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
at
org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
...
"HashSessionScavenger-3" Id=94 TIMED_WAITING on
java.util.TaskQueue@1b1323e7
at java.lang.Object.wait(Native Method)
- waiting on java.util.TaskQueue@1b1323e7
at java.util.TimerThread.mainLoop(Timer.java:552)
at java.util.TimerThread.run(Timer.java:505)
"coreZkRegister-21-thread-1" Id=93 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@59a2094
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@59a2094
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"Thread-34" Id=91 WAITING on java.lang.Object@320a1953
at java.lang.Object.wait(Native Method)
- waiting on java.lang.Object@320a1953
at java.lang.Object.wait(Object.java:503)
at org.apache.solr.core.CloserThread.run(CoreContainer.java:929)
"searcherExecutor-26-thread-1" Id=90 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@49d0d76b
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@49d0d76b
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"TEST-ShardSplitTest.testDistribSearch-seed#[8490788943A65A5D]-EventThread"
Id=87 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2deed7a9
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2deed7a9
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
"TEST-ShardSplitTest.testDistribSearch-seed#[8490788943A65A5D]-SendThread(127.0.0.1:57967)"
Id=86 RUNNABLE (in native)
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
at
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked sun.nio.ch.Util$2@38060b85
- locked java.util.Collections$UnmodifiableSet@14d97f8d
- locked sun.nio.ch.KQueueSelectorImpl@5a431ae9
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
"qtp104014779-82" Id=82 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b9be54a
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b9be54a
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp104014779-81" Id=81 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b9be54a
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b9be54a
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp104014779-80" Id=80 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b9be54a
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b9be54a
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp104014779-79" Id=79 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b9be54a
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b9be54a
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp104014779-78" Id=78 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b9be54a
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b9be54a
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp104014779-77" Id=77 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b9be54a
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b9be54a
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp104014779-76 Acceptor0 [email protected]:57984"
Id=76 RUNNABLE (in native)
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
- locked java.lang.Object@47ea9477
at
org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
at
org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)
"qtp104014779-75 Selector0" Id=75 RUNNABLE (in native)
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
at
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked sun.nio.ch.Util$2@6e6869e3
- locked java.util.Collections$UnmodifiableSet@5a90f357
- locked sun.nio.ch.KQueueSelectorImpl@3a7d5d
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
at
org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
...
"HashSessionScavenger-2" Id=74 TIMED_WAITING on
java.util.TaskQueue@7ffc64c8
at java.lang.Object.wait(Native Method)
- waiting on java.util.TaskQueue@7ffc64c8
at java.util.TimerThread.mainLoop(Timer.java:552)
at java.util.TimerThread.run(Timer.java:505)
"coreZkRegister-12-thread-1" Id=73 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@133cc81d
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@133cc81d
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"Thread-23" Id=71 WAITING on java.lang.Object@320a1953
at java.lang.Object.wait(Native Method)
- waiting on java.lang.Object@320a1953
at java.lang.Object.wait(Object.java:503)
at org.apache.solr.core.CloserThread.run(CoreContainer.java:929)
"searcherExecutor-17-thread-1" Id=70 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7b75f1c8
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7b75f1c8
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"TEST-ShardSplitTest.testDistribSearch-seed#[8490788943A65A5D]-EventThread"
Id=67 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c54c0ca
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c54c0ca
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
"TEST-ShardSplitTest.testDistribSearch-seed#[8490788943A65A5D]-SendThread(127.0.0.1:57967)"
Id=66 RUNNABLE (in native)
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
at
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked sun.nio.ch.Util$2@18adf136
- locked java.util.Collections$UnmodifiableSet@2ac95c29
- locked sun.nio.ch.KQueueSelectorImpl@657d478
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
"qtp2004060070-62" Id=62 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42da41bf
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42da41bf
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp2004060070-61" Id=61 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42da41bf
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42da41bf
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp2004060070-59" Id=59 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42da41bf
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42da41bf
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp2004060070-58" Id=58 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42da41bf
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42da41bf
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp2004060070-57" Id=57 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42da41bf
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42da41bf
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp2004060070-56 Acceptor0 [email protected]:57977"
Id=56 RUNNABLE (in native)
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
- locked java.lang.Object@703e1588
at
org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
at
org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)
"qtp2004060070-55 Selector0" Id=55 RUNNABLE (in native)
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
at
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked sun.nio.ch.Util$2@49c6b334
- locked java.util.Collections$UnmodifiableSet@fd4cc87
- locked sun.nio.ch.KQueueSelectorImpl@89a39b8
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
at
org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
...
"HashSessionScavenger-1" Id=54 TIMED_WAITING on
java.util.TaskQueue@45436371
at java.lang.Object.wait(Native Method)
- waiting on java.util.TaskQueue@45436371
at java.util.TimerThread.mainLoop(Timer.java:552)
at java.util.TimerThread.run(Timer.java:505)
"TEST-ShardSplitTest.testDistribSearch-seed#[8490788943A65A5D]-EventThread"
Id=48 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4b35b33d
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4b35b33d
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
"TEST-ShardSplitTest.testDistribSearch-seed#[8490788943A65A5D]-SendThread(127.0.0.1:57967)"
Id=47 RUNNABLE (in native)
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
at
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked sun.nio.ch.Util$2@270c253e
- locked java.util.Collections$UnmodifiableSet@9d5fa4f
- locked sun.nio.ch.KQueueSelectorImpl@6c7bbfee
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
"coreZkRegister-2-thread-1" Id=46 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3075d9e6
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3075d9e6
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"Thread-12" Id=44 WAITING on java.lang.Object@320a1953
at java.lang.Object.wait(Native Method)
- waiting on java.lang.Object@320a1953
at java.lang.Object.wait(Object.java:503)
at org.apache.solr.core.CloserThread.run(CoreContainer.java:929)
"searcherExecutor-8-thread-1" Id=43 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@934a3e5
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@934a3e5
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"OverseerHdfsCoreFailoverThread-93101408575684611-127.0.0.1:57971__%2Fze-n_0000000000"
Id=41 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at
org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:136)
at java.lang.Thread.run(Thread.java:745)
"OverseerCollectionProcessor-93101408575684611-127.0.0.1:57971__%2Fze-n_0000000000"
Id=40 TIMED_WAITING on java.lang.Object@5ef4141b
at java.lang.Object.wait(Native Method)
- waiting on java.lang.Object@5ef4141b
at
org.apache.solr.cloud.DistributedQueue$LatchWatcher.await(DistributedQueue.java:273)
at
org.apache.solr.cloud.DistributedQueue.getChildren(DistributedQueue.java:317)
at
org.apache.solr.cloud.DistributedQueue.peekTopN(DistributedQueue.java:444)
at
org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:326)
at java.lang.Thread.run(Thread.java:745)
"OverseerStateUpdate-93101408575684611-127.0.0.1:57971__%2Fze-n_0000000000"
Id=39 TIMED_WAITING on java.lang.Object@6a542950
at java.lang.Object.wait(Native Method)
- waiting on java.lang.Object@6a542950
at
org.apache.solr.cloud.DistributedQueue$LatchWatcher.await(DistributedQueue.java:273)
at
org.apache.solr.cloud.DistributedQueue.getChildren(DistributedQueue.java:317)
at
org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:591)
at
org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:569)
at
org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:215)
at java.lang.Thread.run(Thread.java:745)
"TEST-ShardSplitTest.testDistribSearch-seed#[8490788943A65A5D]-EventThread"
Id=37 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b6479b9
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b6479b9
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
"TEST-ShardSplitTest.testDistribSearch-seed#[8490788943A65A5D]-SendThread(127.0.0.1:57967)"
Id=36 RUNNABLE (in native)
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
at
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked sun.nio.ch.Util$2@5a077b5
- locked java.util.Collections$UnmodifiableSet@93b2d3b
- locked sun.nio.ch.KQueueSelectorImpl@197dcb62
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
"qtp1641950186-32" Id=32 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11b39c01
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11b39c01
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp1641950186-31" Id=31 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11b39c01
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11b39c01
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp1641950186-30" Id=30 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11b39c01
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11b39c01
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp1641950186-29" Id=29 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11b39c01
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11b39c01
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp1641950186-28" Id=28 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11b39c01
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11b39c01
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp1641950186-27" Id=27 TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11b39c01
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11b39c01
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp1641950186-26 Acceptor0 [email protected]:57971"
Id=26 RUNNABLE (in native)
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
- locked java.lang.Object@26c70ebb
at
org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
at
org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)
"qtp1641950186-25 Selector0" Id=25 RUNNABLE (in native)
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
at
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked sun.nio.ch.Util$2@2027048a
- locked java.util.Collections$UnmodifiableSet@b593a85
- locked sun.nio.ch.KQueueSelectorImpl@22ee5eed
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
at
org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
...
"HashSessionScavenger-0" Id=24 TIMED_WAITING on
java.util.TaskQueue@4f9a2b85
at java.lang.Object.wait(Native Method)
- waiting on java.util.TaskQueue@4f9a2b85
at java.util.TimerThread.mainLoop(Timer.java:552)
at java.util.TimerThread.run(Timer.java:505)
"ProcessThread(sid:0 cport:-1):" Id=16 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@129f5197
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@129f5197
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:120)
"SyncThread:0" Id=15 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@19a59a0e
at sun.misc.Unsafe.park(Native Method)
- waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@19a59a0e
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:127)
"SessionTracker" Id=14 TIMED_WAITING on
org.apache.zookeeper.server.SessionTrackerImpl@47251a49
at java.lang.Object.wait(Native Method)
- waiting on
org.apache.zookeeper.server.SessionTrackerImpl@47251a49
at
org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146)
"NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0" Id=13 RUNNABLE (in native)
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at
sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
at
sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked sun.nio.ch.Util$2@54e0d31c
- locked java.util.Collections$UnmodifiableSet@7443fecc
- locked sun.nio.ch.KQueueSelectorImpl@1c0d3eb6
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:178)
at java.lang.Thread.run(Thread.java:745)
"Thread-1" Id=12 WAITING on java.lang.Thread@29902d1d
at java.lang.Object.wait(Native Method)
- waiting on java.lang.Thread@29902d1d
at java.lang.Thread.join(Thread.java:1281)
at java.lang.Thread.join(Thread.java:1355)
at
org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:302)
at
org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:303)
at
org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:452)
"TEST-ShardSplitTest.testDistribSearch-seed#[8490788943A65A5D]" Id=11
RUNNABLE
at sun.management.ThreadImpl.dumpThreads0(Native Method)
at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:446)
at
org.apache.solr.core.Diagnostics.logThreadDumps(Diagnostics.java:46)
at
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:176)
at
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:137)
at
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:132)
at
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:823)
at
org.apache.solr.cloud.ShardSplitTest.splitByUniqueKeyTest(ShardSplitTest.java:240)
...
"SUITE-ShardSplitTest-seed#[8490788943A65A5D]" Id=10 TIMED_WAITING on
java.lang.Thread@6ed9b2f9
at java.lang.Object.wait(Native Method)
- waiting on java.lang.Thread@6ed9b2f9
at java.lang.Thread.join(Thread.java:1289)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:803)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:401)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:642)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:129)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$1.run(RandomizedRunner.java:559)
"Monitor Ctrl-Break" Id=8 RUNNABLE (in native)
at java.net.PlainSocketImpl.socketAccept(Native Method)
at
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at
com.intellij.rt.execution.application.AppMain$1.run(AppMain.java:85)
at java.lang.Thread.run(Thread.java:745)
"Signal Dispatcher" Id=4 RUNNABLE
"Finalizer" Id=3 WAITING on java.lang.ref.ReferenceQueue$Lock@3881e03c
at java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.ReferenceQueue$Lock@3881e03c
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
at
java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:189)
"Reference Handler" Id=2 WAITING on
java.lang.ref.Reference$Lock@56a8a01f
at java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.Reference$Lock@56a8a01f
at java.lang.Object.wait(Object.java:503)
at
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
"main" Id=1 WAITING on
com.carrotsearch.randomizedtesting.RandomizedRunner$1@3058b2a9
at java.lang.Object.wait(Native Method)
- waiting on
com.carrotsearch.randomizedtesting.RandomizedRunner$1@3058b2a9
at java.lang.Thread.join(Thread.java:1281)
at java.lang.Thread.join(Thread.java:1355)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:569)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:435)
at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:74)
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:211)
...
{code}
Looking into why are the RecoveryThreads timing out.
> During replication use checksums to verify if files are the same
> ----------------------------------------------------------------
>
> Key: SOLR-6920
> URL: https://issues.apache.org/jira/browse/SOLR-6920
> Project: Solr
> Issue Type: Bug
> Components: replication (java)
> Reporter: Varun Thacker
> Attachments: SOLR-6920.patch
>
>
> Currently we check if an index file on the master and slave is the same by
> checking if it's name and file length match.
> With LUCENE-2446 we now have a checksums for each index file in the segment.
> We should leverage this to verify if two files are the same.
> Places like SnapPuller.isIndexStale and SnapPuller.downloadIndexFiles should
> check against the checksum also.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
