See <http://sierranevada.servebeer.com:8080/job/slow-io-beasting/5020/>
------------------------------------------ [...truncated 14006 lines...] [junit4:junit4] 2> 690757 T282 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:58646 which had sessionid 0x13aa99d68d00002 [junit4:junit4] 2> 690757 T282 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:58708 which had sessionid 0x13aa99d68d00004 [junit4:junit4] 2> 690757 T303 oaz.ClientCnxn$SendThread.run Unable to read additional data from server sessionid 0x13aa99d68d00002, likely server has closed socket, closing socket connection and attempting reconnect [junit4:junit4] 2> 690757 T331 oaz.ClientCnxn$SendThread.run Unable to read additional data from server sessionid 0x13aa99d68d00004, likely server has closed socket, closing socket connection and attempting reconnect [junit4:junit4] 2> 690757 T282 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:59242 which had sessionid 0x13aa99d68d00006 [junit4:junit4] 2> 690757 T355 oaz.ClientCnxn$SendThread.run Unable to read additional data from server sessionid 0x13aa99d68d00006, likely server has closed socket, closing socket connection and attempting reconnect [junit4:junit4] 2> 690757 T282 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:58739 which had sessionid 0x13aa99d68d00005 [junit4:junit4] 2> 690757 T343 oaz.ClientCnxn$SendThread.run Unable to read additional data from server sessionid 0x13aa99d68d00005, likely server has closed socket, closing socket connection and attempting reconnect [junit4:junit4] 2> 690757 T282 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:58674 which had sessionid 0x13aa99d68d00003 [junit4:junit4] 2> 690757 T284 oazs.NIOServerCnxn$Factory.run NIOServerCnxn factory exited run method [junit4:junit4] 2> 690757 T317 oaz.ClientCnxn$SendThread.run Unable to read additional data from server sessionid 0x13aa99d68d00003, likely server has closed socket, closing socket connection and attempting reconnect [junit4:junit4] 2> 690757 T282 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete [junit4:junit4] 2> 690757 T282 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 58639 [junit4:junit4] 2> 690757 T282 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=25232942 [junit4:junit4] 2> 690757 T282 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@7e4f51 [junit4:junit4] 2> 690757 T282 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0} [junit4:junit4] 2> 690757 T282 oasc.SolrCore.decrefSolrCoreState Closing SolrCoreState [junit4:junit4] 2> 690757 T282 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter [junit4:junit4] 2> 690757 T282 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser [junit4:junit4] 2> 690757 T282 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request. [junit4:junit4] 2> 690757 T305 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=88569636220567554-127.0.0.1:58639_solr-n_0000000000) am no longer a leader. [junit4:junit4] 2> 690866 T318 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4985d9 name:ZooKeeperConnection Watcher:127.0.0.1:58626/solr got event WatchedEvent state:Disconnected type:None path:null path:null type:None [junit4:junit4] 2> 690866 T344 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13c22b0 name:ZooKeeperConnection Watcher:127.0.0.1:58626/solr got event WatchedEvent state:Disconnected type:None path:null path:null type:None [junit4:junit4] 2> 690866 T356 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b66ab4 name:ZooKeeperConnection Watcher:127.0.0.1:58626/solr got event WatchedEvent state:Disconnected type:None path:null path:null type:None [junit4:junit4] 2> 690866 T318 oascc.ConnectionManager.process zkClient has disconnected [junit4:junit4] 2> 690866 T344 oascc.ConnectionManager.process zkClient has disconnected [junit4:junit4] 2> 690866 T356 oascc.ConnectionManager.process zkClient has disconnected [junit4:junit4] 2> 690866 T332 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b57ca1 name:ZooKeeperConnection Watcher:127.0.0.1:58626/solr got event WatchedEvent state:Disconnected type:None path:null path:null type:None [junit4:junit4] 2> 690866 T332 oascc.ConnectionManager.process zkClient has disconnected [junit4:junit4] 2> 690866 T304 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14a65f8 name:ZooKeeperConnection Watcher:127.0.0.1:58626/solr got event WatchedEvent state:Disconnected type:None path:null path:null type:None [junit4:junit4] 2> 690866 T282 oaz.ZooKeeper.close Session: 0x13aa99d68d00002 closed [junit4:junit4] 2> 690866 T304 oascc.ConnectionManager.process Client->ZooKeeper status change trigger but we are already closed [junit4:junit4] 2> 690866 T304 oaz.ClientCnxn$EventThread.run EventThread shut down [junit4:junit4] 2> 690867 T282 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null} [junit4:junit4] 2> 690918 T282 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 58669 [junit4:junit4] 2> 690918 T282 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=2908606 [junit4:junit4] 2> 690918 T282 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@d75f7a [junit4:junit4] 2> 690922 T282 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0} [junit4:junit4] 2> 690922 T282 oasc.SolrCore.decrefSolrCoreState Closing SolrCoreState [junit4:junit4] 2> 690922 T282 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter [junit4:junit4] 2> 690923 T282 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser [junit4:junit4] 2> 690923 T282 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request. [junit4:junit4] 2> 692065 T343 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server 127.0.0.1/127.0.0.1:58626 [junit4:junit4] 2> 692236 T317 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server 127.0.0.1/127.0.0.1:58626 [junit4:junit4] 2> 692580 T331 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server 127.0.0.1/127.0.0.1:58626 [junit4:junit4] 2> 692845 T355 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server 127.0.0.1/127.0.0.1:58626 [junit4:junit4] 2> 693063 T343 oaz.ClientCnxn$SendThread.run WARNING Session 0x13aa99d68d00005 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused: no further information [junit4:junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [junit4:junit4] 2> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567) [junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143) [junit4:junit4] 2> [junit4:junit4] 2> 693314 T318 oaz.ClientCnxn$EventThread.run EventThread shut down [junit4:junit4] 2> 693314 T282 oaz.ZooKeeper.close Session: 0x13aa99d68d00003 closed [junit4:junit4] 2> 693325 T282 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null} [junit4:junit4] 2> 693376 T282 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 58703 [junit4:junit4] 2> 693376 T282 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=15813854 [junit4:junit4] 2> 693376 T282 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@6b55c1 [junit4:junit4] 2> 693380 T282 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0} [junit4:junit4] 2> 693380 T282 oasc.SolrCore.decrefSolrCoreState Closing SolrCoreState [junit4:junit4] 2> 693381 T282 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter [junit4:junit4] 2> 693381 T282 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser [junit4:junit4] 2> 693383 T282 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request. [junit4:junit4] 2> 693634 T332 oaz.ClientCnxn$EventThread.run EventThread shut down [junit4:junit4] 2> 693634 T282 oaz.ZooKeeper.close Session: 0x13aa99d68d00004 closed [junit4:junit4] 2> 693644 T282 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null} [junit4:junit4] 2> 693695 T282 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 58734 [junit4:junit4] 2> 693695 T282 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=18062375 [junit4:junit4] 2> 693814 T355 oaz.ClientCnxn$SendThread.run WARNING Session 0x13aa99d68d00006 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused: no further information [junit4:junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [junit4:junit4] 2> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567) [junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143) [junit4:junit4] 2> [junit4:junit4] 2> 695169 T355 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server 127.0.0.1/127.0.0.1:58626 [junit4:junit4] 2> 695169 T343 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server 127.0.0.1/127.0.0.1:58626 [junit4:junit4] 2> 696183 T355 oaz.ClientCnxn$SendThread.run WARNING Session 0x13aa99d68d00006 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused: no further information [junit4:junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [junit4:junit4] 2> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567) [junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143) [junit4:junit4] 2> [junit4:junit4] 2> 696292 T344 oaz.ClientCnxn$EventThread.run EventThread shut down [junit4:junit4] 2> 696292 T282 oaz.ZooKeeper.close Session: 0x13aa99d68d00005 closed [junit4:junit4] 2> 696293 T282 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null} [junit4:junit4] 2> 696344 T282 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 59235 [junit4:junit4] 2> 696344 T282 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=33028132 [junit4:junit4] 2> 696344 T282 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@14c2b2f [junit4:junit4] 2> 696353 T282 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0} [junit4:junit4] 2> 696354 T282 oasc.SolrCore.decrefSolrCoreState Closing SolrCoreState [junit4:junit4] 2> 696354 T282 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter [junit4:junit4] 2> 696354 T282 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser [junit4:junit4] 2> 696356 T282 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request. [junit4:junit4] 2> 698109 T355 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server 127.0.0.1/127.0.0.1:58626 [junit4:junit4] 2> 699232 T356 oaz.ClientCnxn$EventThread.run EventThread shut down [junit4:junit4] 2> 699232 T282 oaz.ZooKeeper.close Session: 0x13aa99d68d00006 closed [junit4:junit4] 2> 699248 T282 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null} [junit4:junit4] 2> 699501 T282 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch [junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=FullSolrCloudDistribCmdsTest -Dtests.method=testDistribSearch -Dtests.seed=F1B077980FD36247 -Dtests.slow=true -Dtests.locale=bg -Dtests.timezone=Indian/Mahe -Dtests.file.encoding=ISO-8859-1 [junit4:junit4] FAILURE 700s J4 | FullSolrCloudDistribCmdsTest.testDistribSearch <<< [junit4:junit4] > Throwable #1: java.lang.AssertionError: There are still nodes recoverying - waited for 600 seconds [junit4:junit4] > at __randomizedtesting.SeedInfo.seed([F1B077980FD36247:7056F980788C027B]:0) [junit4:junit4] > at org.junit.Assert.fail(Assert.java:93) [junit4:junit4] > at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:156) [junit4:junit4] > at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:115) [junit4:junit4] > at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:110) [junit4:junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:553) [junit4:junit4] > at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.doTest(FullSolrCloudDistribCmdsTest.java:65) [junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:695) [junit4:junit4] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [junit4:junit4] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [junit4:junit4] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [junit4:junit4] > at java.lang.reflect.Method.invoke(Method.java:597) [junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559) [junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79) [junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737) [junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773) [junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787) [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) [junit4:junit4] > at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) [junit4:junit4] > at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51) [junit4:junit4] > at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) [junit4:junit4] > at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) [junit4:junit4] > at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70) [junit4:junit4] > at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358) [junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782) [junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442) [junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746) [junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648) [junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682) [junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693) [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) [junit4:junit4] > at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [junit4:junit4] > at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4:junit4] > at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43) [junit4:junit4] > at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) [junit4:junit4] > at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70) [junit4:junit4] > at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358) [junit4:junit4] > at java.lang.Thread.run(Thread.java:662) [junit4:junit4] 2> 699517 T282 oas.SolrTestCaseJ4.deleteCore ###deleteCore [junit4:junit4] 2> 822211 T282 oas.SolrTestCaseJ4.endTrackingSearchers SEVERE ERROR: SolrIndexSearcher opens=7 closes=6 [junit4:junit4] 2> 822211 T281 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s). [junit4:junit4] 2> 842366 T281 ccr.ThreadLeakControl.checkThreadLeaks SEVERE 1 thread leaked from SUITE scope at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest: [junit4:junit4] 2> 1) Thread[id=345, name=searcherExecutor-163-thread-1, state=WAITING, group=TGRP-FullSolrCloudDistribCmdsTest] [junit4:junit4] 2> at sun.misc.Unsafe.park(Native Method) [junit4:junit4] 2> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) [junit4:junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) [junit4:junit4] 2> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) [junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) [junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) [junit4:junit4] 2> at java.lang.Thread.run(Thread.java:662) [junit4:junit4] 2> 842366 T281 ccr.ThreadLeakControl.tryToInterruptAll Starting to interrupt leaked threads: [junit4:junit4] 2> 1) Thread[id=345, name=searcherExecutor-163-thread-1, state=WAITING, group=TGRP-FullSolrCloudDistribCmdsTest] [junit4:junit4] 2> 845455 T281 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated: [junit4:junit4] 2> 1) Thread[id=345, name=searcherExecutor-163-thread-1, state=WAITING, group=TGRP-FullSolrCloudDistribCmdsTest] [junit4:junit4] 2> at sun.misc.Unsafe.park(Native Method) [junit4:junit4] 2> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) [junit4:junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) [junit4:junit4] 2> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) [junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) [junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) [junit4:junit4] 2> at java.lang.Thread.run(Thread.java:662) [junit4:junit4] 2> NOTE: test params are: codec=Asserting, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=bg, timezone=Indian/Mahe [junit4:junit4] 2> NOTE: Windows Vista 6.0 x86/Sun Microsystems Inc. 1.6.0_23 (32-bit)/cpus=4,threads=2,free=6564744,total=32903168 [junit4:junit4] 2> NOTE: All tests run in this JVM: [XsltUpdateRequestHandlerTest, TestDocSet, TermVectorComponentTest, SolrCmdDistributorTest, TestPseudoReturnFields, URLClassifyProcessorTest, NoCacheHeaderTest, TestSystemIdResolver, FileBasedSpellCheckerTest, TestIndexSearcher, TestStressReorder, FastVectorHighlighterTest, TestLMJelinekMercerSimilarityFactory, StatsComponentTest, ScriptEngineTest, LukeRequestHandlerTest, FullSolrCloudDistribCmdsTest] [junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=FullSolrCloudDistribCmdsTest -Dtests.seed=F1B077980FD36247 -Dtests.slow=true -Dtests.locale=bg -Dtests.timezone=Indian/Mahe -Dtests.file.encoding=ISO-8859-1 [junit4:junit4] ERROR 0.00s J4 | FullSolrCloudDistribCmdsTest (suite) <<< [junit4:junit4] > Throwable #1: java.lang.AssertionError: ERROR: SolrIndexSearcher opens=7 closes=6 [junit4:junit4] > at __randomizedtesting.SeedInfo.seed([F1B077980FD36247]:0) [junit4:junit4] > at org.junit.Assert.fail(Assert.java:93) [junit4:junit4] > at org.apache.solr.SolrTestCaseJ4.endTrackingSearchers(SolrTestCaseJ4.java:242) [junit4:junit4] > at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:97) [junit4:junit4] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [junit4:junit4] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [junit4:junit4] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [junit4:junit4] > at java.lang.reflect.Method.invoke(Method.java:597) [junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559) [junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79) [junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:700) [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) [junit4:junit4] > at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [junit4:junit4] > at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4:junit4] > at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43) [junit4:junit4] > at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) [junit4:junit4] > at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70) [junit4:junit4] > at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358) [junit4:junit4] > at java.lang.Thread.run(Thread.java:662) [junit4:junit4] > Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest: [junit4:junit4] > 1) Thread[id=345, name=searcherExecutor-163-thread-1, state=WAITING, group=TGRP-FullSolrCloudDistribCmdsTest] [junit4:junit4] > at sun.misc.Unsafe.park(Native Method) [junit4:junit4] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) [junit4:junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) [junit4:junit4] > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) [junit4:junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) [junit4:junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) [junit4:junit4] > at java.lang.Thread.run(Thread.java:662) [junit4:junit4] > at __randomizedtesting.SeedInfo.seed([F1B077980FD36247]:0) [junit4:junit4] > Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated: [junit4:junit4] > 1) Thread[id=345, name=searcherExecutor-163-thread-1, state=WAITING, group=TGRP-FullSolrCloudDistribCmdsTest] [junit4:junit4] > at sun.misc.Unsafe.park(Native Method) [junit4:junit4] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) [junit4:junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) [junit4:junit4] > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) [junit4:junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) [junit4:junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) [junit4:junit4] > at java.lang.Thread.run(Thread.java:662) [junit4:junit4] > at __randomizedtesting.SeedInfo.seed([F1B077980FD36247]:0) [junit4:junit4] Completed on J4 in 845.50s, 1 test, 2 failures, 2 errors <<< FAILURES! [junit4:junit4] [junit4:junit4] [junit4:junit4] Tests with failures: [junit4:junit4] - org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testDistribSearch [junit4:junit4] - org.apache.solr.cloud.FullSolrCloudDistribCmdsTest (suite) [junit4:junit4] [junit4:junit4] [junit4:junit4] JVM J0: 2.62 .. 273.28 = 270.66s [junit4:junit4] JVM J1: 2.62 .. 269.64 = 267.02s [junit4:junit4] JVM J2: 2.87 .. 341.30 = 338.43s [junit4:junit4] JVM J3: 2.62 .. 270.27 = 267.65s [junit4:junit4] JVM J4: 3.40 .. 974.88 = 971.47s [junit4:junit4] JVM J5: 2.62 .. 280.39 = 277.77s [junit4:junit4] JVM J6: 2.62 .. 275.62 = 273.00s [junit4:junit4] JVM J7: 2.62 .. 270.00 = 267.38s [junit4:junit4] Execution time total: 16 minutes 14 seconds [junit4:junit4] Tests summary: 233 suites, 965 tests, 3 suite-level errors, 1 failure, 13 ignored (8 assumptions) BUILD FAILED <http://sierranevada.servebeer.com:8080/job/slow-io-beasting/ws/build.xml>:39: The following error occurred while executing this line: <http://sierranevada.servebeer.com:8080/job/slow-io-beasting/ws/solr/build.xml>:178: The following error occurred while executing this line: <http://sierranevada.servebeer.com:8080/job/slow-io-beasting/ws/lucene/module-build.xml>:61: The following error occurred while executing this line: <http://sierranevada.servebeer.com:8080/job/slow-io-beasting/ws/lucene/common-build.xml>:1159: The following error occurred while executing this line: <http://sierranevada.servebeer.com:8080/job/slow-io-beasting/ws/lucene/common-build.xml>:827: There were test failures: 233 suites, 965 tests, 3 suite-level errors, 1 failure, 13 ignored (8 assumptions) Total time: 24 minutes 34 seconds Build step 'Invoke Ant' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org