I'll look into this. D.
On Wed, Jul 25, 2018 at 6:12 AM Apache Jenkins Server
<jenk...@builds.apache.org> wrote:
>
> Build: 
> https://builds.apache.org/job/Lucene-Solr-BadApples-NightlyTests-7.x/19/
>
> 16 tests failed.
> FAILED:  
> org.apache.lucene.store.TestMockDirectoryWrapper.testThreadSafetyInListAll
>
> Error Message:
> Captured an uncaught exception in thread: Thread[id=30, name=Thread-8, 
> state=RUNNABLE, group=TGRP-TestMockDirectoryWrapper]
>
> Stack Trace:
> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an 
> uncaught exception in thread: Thread[id=30, name=Thread-8, state=RUNNABLE, 
> group=TGRP-TestMockDirectoryWrapper]
>         at 
> __randomizedtesting.SeedInfo.seed([3F18BAEF97C7B4BC:24042533623F21CB]:0)
> Caused by: java.io.UncheckedIOException: Something went wrong when opening: 
> file-29
>         at __randomizedtesting.SeedInfo.seed([3F18BAEF97C7B4BC]:0)
>         at 
> org.apache.lucene.store.BaseDirectoryTestCase.lambda$testThreadSafetyInListAll$11(BaseDirectoryTestCase.java:446)
>         at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException: MockDirectoryWrapper: file "file-29" is still 
> open for writing
>         at 
> org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:752)
>         at 
> org.apache.lucene.store.BaseDirectoryTestCase.lambda$testThreadSafetyInListAll$11(BaseDirectoryTestCase.java:440)
>         ... 1 more
> Caused by: java.lang.RuntimeException: unclosed IndexOutput: file-29
>         at 
> org.apache.lucene.store.MockDirectoryWrapper.addFileHandle(MockDirectoryWrapper.java:729)
>         at 
> org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:668)
>         at 
> org.apache.lucene.store.BaseDirectoryTestCase.lambda$testThreadSafetyInListAll$8(BaseDirectoryTestCase.java:416)
>         ... 1 more
>
>
> FAILED:  org.apache.solr.cloud.BasicDistributedZkTest.test
>
> Error Message:
> Error from server at http://127.0.0.1:35231/mf_xod/lg/collection2: No 
> registered leader was found after waiting for 4000ms , collection: 
> collection2 slice: shard1 saw 
> state=DocCollection(collection2//collections/collection2/state.json/6)={   
> "pullReplicas":"0",   "replicationFactor":"1",   "shards":{     "shard1":{    
>    "range":"80000000-ffffffff",       "state":"active",       "replicas":{}}, 
>     "shard2":{       "range":"0-7fffffff",       "state":"active",       
> "replicas":{         "core_node42":{           
> "core":"collection2_shard2_replica_n41",           
> "base_url":"http://127.0.0.1:35231/mf_xod/lg";,           
> "node_name":"127.0.0.1:35231_mf_xod%2Flg",           "state":"active",        
>    "type":"NRT",           "leader":"true"},         "core_node45":{          
>  "core":"collection2_shard2_replica_n44",           
> "base_url":"http://127.0.0.1:34642/mf_xod/lg";,           
> "node_name":"127.0.0.1:34642_mf_xod%2Flg",           "state":"active",        
>    "type":"NRT"}}}},   "router":{"name":"compositeId"},   
> "maxShardsPerNode":"1",   "autoAddReplicas":"false",   "nrtReplicas":"1",   
> "tlogReplicas":"0"} with live_nodes=[127.0.0.1:34747_mf_xod%2Flg, 
> 127.0.0.1:45188_mf_xod%2Flg, 127.0.0.1:34642_mf_xod%2Flg, 
> 127.0.0.1:33906_mf_xod%2Flg, 127.0.0.1:35231_mf_xod%2Flg]
>
> Stack Trace:
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
> from server at http://127.0.0.1:35231/mf_xod/lg/collection2: No registered 
> leader was found after waiting for 4000ms , collection: collection2 slice: 
> shard1 saw 
> state=DocCollection(collection2//collections/collection2/state.json/6)={
>   "pullReplicas":"0",
>   "replicationFactor":"1",
>   "shards":{
>     "shard1":{
>       "range":"80000000-ffffffff",
>       "state":"active",
>       "replicas":{}},
>     "shard2":{
>       "range":"0-7fffffff",
>       "state":"active",
>       "replicas":{
>         "core_node42":{
>           "core":"collection2_shard2_replica_n41",
>           "base_url":"http://127.0.0.1:35231/mf_xod/lg";,
>           "node_name":"127.0.0.1:35231_mf_xod%2Flg",
>           "state":"active",
>           "type":"NRT",
>           "leader":"true"},
>         "core_node45":{
>           "core":"collection2_shard2_replica_n44",
>           "base_url":"http://127.0.0.1:34642/mf_xod/lg";,
>           "node_name":"127.0.0.1:34642_mf_xod%2Flg",
>           "state":"active",
>           "type":"NRT"}}}},
>   "router":{"name":"compositeId"},
>   "maxShardsPerNode":"1",
>   "autoAddReplicas":"false",
>   "nrtReplicas":"1",
>   "tlogReplicas":"0"} with live_nodes=[127.0.0.1:34747_mf_xod%2Flg, 
> 127.0.0.1:45188_mf_xod%2Flg, 127.0.0.1:34642_mf_xod%2Flg, 
> 127.0.0.1:33906_mf_xod%2Flg, 127.0.0.1:35231_mf_xod%2Flg]
>         at 
> __randomizedtesting.SeedInfo.seed([EE9E6F4F6108C35B:66CA5095CFF4AEA3]:0)
>         at 
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
>         at 
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
>         at 
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
>         at 
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
>         at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173)
>         at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138)
>         at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:152)
>         at 
> org.apache.solr.cloud.BasicDistributedZkTest.indexDoc(BasicDistributedZkTest.java:1081)
>         at 
> org.apache.solr.cloud.BasicDistributedZkTest.testMultipleCollections(BasicDistributedZkTest.java:1023)
>         at 
> org.apache.solr.cloud.BasicDistributedZkTest.test(BasicDistributedZkTest.java:374)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at 
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
>         at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
>         at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
>         at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
>         at 
> org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1008)
>         at 
> org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:983)
>         at 
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>         at 
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
>         at 
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>         at 
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
>         at 
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>         at 
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>         at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at 
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>         at 
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
>         at 
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
>         at 
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
>         at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
>         at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
>         at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
>         at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at 
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>         at 
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>         at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at 
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>         at 
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>         at 
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>         at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at 
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>         at 
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>         at 
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>         at 
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
>         at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at 
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>         at java.lang.Thread.run(Thread.java:748)
>
>
> FAILED:  
> org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testSplitIntegration
>
> Error Message:
> last state: 
> DocCollection(testSplitIntegration_collection//clusterstate.json/107)={   
> "replicationFactor":"1",   "pullReplicas":"0",   
> "router":{"name":"compositeId"},   "maxShardsPerNode":"2",   
> "autoAddReplicas":"false",   "nrtReplicas":"2",   "tlogReplicas":"0",   
> "autoCreated":"true",   "shards":{     "shard2":{       "replicas":{         
> "core_node3":{           
> "core":"testSplitIntegration_collection_shard2_replica_n3",           
> "leader":"true",           "SEARCHER.searcher.maxDoc":11,           
> "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":15740,       
>     "node_name":"127.0.0.1:10000_solr",           "state":"active",           
> "type":"NRT",           "INDEX.sizeInGB":1.4659017324447632E-5,           
> "SEARCHER.searcher.numDocs":11},         "core_node4":{           
> "core":"testSplitIntegration_collection_shard2_replica_n4",           
> "SEARCHER.searcher.maxDoc":11,           "SEARCHER.searcher.deletedDocs":0,   
>         "INDEX.sizeInBytes":15740,           
> "node_name":"127.0.0.1:10001_solr",           "state":"active",           
> "type":"NRT",           "INDEX.sizeInGB":1.4659017324447632E-5,           
> "SEARCHER.searcher.numDocs":11}},       "range":"0-7fffffff",       
> "state":"active"},     "shard1":{       
> "stateTimestamp":"1532487213672955450",       "replicas":{         
> "core_node1":{           
> "core":"testSplitIntegration_collection_shard1_replica_n1",           
> "leader":"true",           "SEARCHER.searcher.maxDoc":14,           
> "SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":17240,       
>     "node_name":"127.0.0.1:10000_solr",           "state":"active",           
> "type":"NRT",           "INDEX.sizeInGB":1.605600118637085E-5,           
> "SEARCHER.searcher.numDocs":14},         "core_node2":{           
> "core":"testSplitIntegration_collection_shard1_replica_n2",           
> "SEARCHER.searcher.maxDoc":14,           "SEARCHER.searcher.deletedDocs":0,   
>         "INDEX.sizeInBytes":17240,           
> "node_name":"127.0.0.1:10001_solr",           "state":"active",           
> "type":"NRT",           "INDEX.sizeInGB":1.605600118637085E-5,           
> "SEARCHER.searcher.numDocs":14}},       "range":"80000000-ffffffff",       
> "state":"inactive"},     "shard1_1":{       "parent":"shard1",       
> "stateTimestamp":"1532487213673722550",       "range":"c0000000-ffffffff",    
>    "state":"active",       "replicas":{         "core_node10":{           
> "leader":"true",           
> "core":"testSplitIntegration_collection_shard1_1_replica1",           
> "SEARCHER.searcher.maxDoc":7,           "SEARCHER.searcher.deletedDocs":0,    
>        "INDEX.sizeInBytes":13740,           
> "node_name":"127.0.0.1:10000_solr",           
> "base_url":"http://127.0.0.1:10000/solr";,           "state":"active",         
>   "type":"NRT",           "INDEX.sizeInGB":1.2796372175216675E-5,           
> "SEARCHER.searcher.numDocs":7},         "core_node9":{           
> "core":"testSplitIntegration_collection_shard1_1_replica0",           
> "SEARCHER.searcher.maxDoc":7,           "SEARCHER.searcher.deletedDocs":0,    
>        "INDEX.sizeInBytes":13740,           
> "node_name":"127.0.0.1:10001_solr",           
> "base_url":"http://127.0.0.1:10001/solr";,           "state":"active",         
>   "type":"NRT",           "INDEX.sizeInGB":1.2796372175216675E-5,           
> "SEARCHER.searcher.numDocs":7}}},     "shard1_0":{       "parent":"shard1",   
>     "stateTimestamp":"1532487213673606150",       
> "range":"80000000-bfffffff",       "state":"active",       "replicas":{       
>   "core_node7":{           "leader":"true",           
> "core":"testSplitIntegration_collection_shard1_0_replica0",           
> "SEARCHER.searcher.maxDoc":7,           "SEARCHER.searcher.deletedDocs":0,    
>        "INDEX.sizeInBytes":23980,           
> "node_name":"127.0.0.1:10001_solr",           
> "base_url":"http://127.0.0.1:10001/solr";,           "state":"active",         
>   "type":"NRT",           "INDEX.sizeInGB":2.2333115339279175E-5,           
> "SEARCHER.searcher.numDocs":7},         "core_node8":{           
> "core":"testSplitIntegration_collection_shard1_0_replica1",           
> "SEARCHER.searcher.maxDoc":7,           "SEARCHER.searcher.deletedDocs":0,    
>        "INDEX.sizeInBytes":23980,           
> "node_name":"127.0.0.1:10000_solr",           
> "base_url":"http://127.0.0.1:10000/solr";,           "state":"active",         
>   "type":"NRT",           "INDEX.sizeInGB":2.2333115339279175E-5,           
> "SEARCHER.searcher.numDocs":7}}}}}
>
> Stack Trace:
> java.util.concurrent.TimeoutException: last state: 
> DocCollection(testSplitIntegration_collection//clusterstate.json/107)={
>   "replicationFactor":"1",
>   "pullReplicas":"0",
>   "router":{"name":"compositeId"},
>   "maxShardsPerNode":"2",
>   "autoAddReplicas":"false",
>   "nrtReplicas":"2",
>   "tlogReplicas":"0",
>   "autoCreated":"true",
>   "shards":{
>     "shard2":{
>       "replicas":{
>         "core_node3":{
>           "core":"testSplitIntegration_collection_shard2_replica_n3",
>           "leader":"true",
>           "SEARCHER.searcher.maxDoc":11,
>           "SEARCHER.searcher.deletedDocs":0,
>           "INDEX.sizeInBytes":15740,
>           "node_name":"127.0.0.1:10000_solr",
>           "state":"active",
>           "type":"NRT",
>           "INDEX.sizeInGB":1.4659017324447632E-5,
>           "SEARCHER.searcher.numDocs":11},
>         "core_node4":{
>           "core":"testSplitIntegration_collection_shard2_replica_n4",
>           "SEARCHER.searcher.maxDoc":11,
>           "SEARCHER.searcher.deletedDocs":0,
>           "INDEX.sizeInBytes":15740,
>           "node_name":"127.0.0.1:10001_solr",
>           "state":"active",
>           "type":"NRT",
>           "INDEX.sizeInGB":1.4659017324447632E-5,
>           "SEARCHER.searcher.numDocs":11}},
>       "range":"0-7fffffff",
>       "state":"active"},
>     "shard1":{
>       "stateTimestamp":"1532487213672955450",
>       "replicas":{
>         "core_node1":{
>           "core":"testSplitIntegration_collection_shard1_replica_n1",
>           "leader":"true",
>           "SEARCHER.searcher.maxDoc":14,
>           "SEARCHER.searcher.deletedDocs":0,
>           "INDEX.sizeInBytes":17240,
>           "node_name":"127.0.0.1:10000_solr",
>           "state":"active",
>           "type":"NRT",
>           "INDEX.sizeInGB":1.605600118637085E-5,
>           "SEARCHER.searcher.numDocs":14},
>         "core_node2":{
>           "core":"testSplitIntegration_collection_shard1_replica_n2",
>           "SEARCHER.searcher.maxDoc":14,
>           "SEARCHER.searcher.deletedDocs":0,
>           "INDEX.sizeInBytes":17240,
>           "node_name":"127.0.0.1:10001_solr",
>           "state":"active",
>           "type":"NRT",
>           "INDEX.sizeInGB":1.605600118637085E-5,
>           "SEARCHER.searcher.numDocs":14}},
>       "range":"80000000-ffffffff",
>       "state":"inactive"},
>     "shard1_1":{
>       "parent":"shard1",
>       "stateTimestamp":"1532487213673722550",
>       "range":"c0000000-ffffffff",
>       "state":"active",
>       "replicas":{
>         "core_node10":{
>           "leader":"true",
>           "core":"testSplitIntegration_collection_shard1_1_replica1",
>           "SEARCHER.searcher.maxDoc":7,
>           "SEARCHER.searcher.deletedDocs":0,
>           "INDEX.sizeInBytes":13740,
>           "node_name":"127.0.0.1:10000_solr",
>           "base_url":"http://127.0.0.1:10000/solr";,
>           "state":"active",
>           "type":"NRT",
>           "INDEX.sizeInGB":1.2796372175216675E-5,
>           "SEARCHER.searcher.numDocs":7},
>         "core_node9":{
>           "core":"testSplitIntegration_collection_shard1_1_replica0",
>           "SEARCHER.searcher.maxDoc":7,
>           "SEARCHER.searcher.deletedDocs":0,
>           "INDEX.sizeInBytes":13740,
>           "node_name":"127.0.0.1:10001_solr",
>           "base_url":"http://127.0.0.1:10001/solr";,
>           "state":"active",
>           "type":"NRT",
>           "INDEX.sizeInGB":1.2796372175216675E-5,
>           "SEARCHER.searcher.numDocs":7}}},
>     "shard1_0":{
>       "parent":"shard1",
>       "stateTimestamp":"1532487213673606150",
>       "range":"80000000-bfffffff",
>       "state":"active",
>       "replicas":{
>         "core_node7":{
>           "leader":"true",
>           "core":"testSplitIntegration_collection_shard1_0_replica0",
>           "SEARCHER.searcher.maxDoc":7,
>           "SEARCHER.searcher.deletedDocs":0,
>           "INDEX.sizeInBytes":23980,
>           "node_name":"127.0.0.1:10001_solr",
>           "base_url":"http://127.0.0.1:10001/solr";,
>           "state":"active",
>           "type":"NRT",
>           "INDEX.sizeInGB":2.2333115339279175E-5,
>           "SEARCHER.searcher.numDocs":7},
>         "core_node8":{
>           "core":"testSplitIntegration_collection_shard1_0_replica1",
>           "SEARCHER.searcher.maxDoc":7,
>           "SEARCHER.searcher.deletedDocs":0,
>           "INDEX.sizeInBytes":23980,
>           "node_name":"127.0.0.1:10000_solr",
>           "base_url":"http://127.0.0.1:10000/solr";,
>           "state":"active",
>           "type":"NRT",
>           "INDEX.sizeInGB":2.2333115339279175E-5,
>           "SEARCHER.searcher.numDocs":7}}}}}
>         at 
> __randomizedtesting.SeedInfo.seed([EE9E6F4F6108C35B:D710D60F4EF70AA5]:0)
>         at 
> org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:111)
>         at 
> org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testSplitIntegration(IndexSizeTriggerTest.java:302)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at 
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
>         at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
>         at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
>         at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
>         at 
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>         at 
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
>         at 
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>         at 
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
>         at 
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>         at 
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>         at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at 
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>         at 
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
>         at 
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
>         at 
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
>         at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
>         at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
>         at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
>         at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at 
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>         at 
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>         at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at 
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>         at 
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>         at 
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>         at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at 
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>         at 
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>         at 
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>         at 
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
>         at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at 
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>         at java.lang.Thread.run(Thread.java:748)
>
>
> FAILED:  
> org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testMergeIntegration
>
> Error Message:
> did not finish processing in time
>
> Stack Trace:
> java.lang.AssertionError: did not finish processing in time
>         at 
> __randomizedtesting.SeedInfo.seed([EE9E6F4F6108C35B:BD272DFF831956A1]:0)
>         at org.junit.Assert.fail(Assert.java:93)
>         at org.junit.Assert.assertTrue(Assert.java:43)
>         at 
> org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testMergeIntegration(IndexSizeTriggerTest.java:408)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at 
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
>         at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
>         at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
>         at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
>         at 
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>         at 
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
>         at 
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>         at 
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
>         at 
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>         at 
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>         at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at 
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>         at 
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
>         at 
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
>         at 
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
>         at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
>         at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
>         at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
>         at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at 
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>         at 
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>         at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at 
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>         at 
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>         at 
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>         at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at 
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>         at 
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>         at 
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>         at 
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
>         at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at 
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>         at java.lang.Thread.run(Thread.java:748)
>
>
> FAILED:  
> junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest
>
> Error Message:
> 4 threads leaked from SUITE scope at 
> org.apache.solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest:     1) 
> Thread[id=9359, name=Connection evictor, state=TIMED_WAITING, 
> group=TGRP-HdfsChaosMonkeyNothingIsSafeTest]         at 
> java.lang.Thread.sleep(Native Method)         at 
> org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
>          at java.lang.Thread.run(Thread.java:748)    2) Thread[id=9360, 
> name=TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[EE9E6F4F6108C35B]-SendThread(127.0.0.1:39603),
>  state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeyNothingIsSafeTest]         at 
> java.lang.Thread.sleep(Native Method)         at 
> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)
>          at 
> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000) 
>         at 
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)    3) 
> Thread[id=9361, 
> name=TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[EE9E6F4F6108C35B]-EventThread,
>  state=WAITING, group=TGRP-HdfsChaosMonkeyNothingIsSafeTest]         at 
> sun.misc.Unsafe.park(Native Method)         at 
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>          at 
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)   
>       at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502) 
>    4) Thread[id=9362, name=zkConnectionManagerCallback-4269-thread-1, 
> state=WAITING, group=TGRP-HdfsChaosMonkeyNothingIsSafeTest]         at 
> sun.misc.Unsafe.park(Native Method)         at 
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>          at 
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)   
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) 
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>          at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>          at java.lang.Thread.run(Thread.java:748)
>
> Stack Trace:
> com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from 
> SUITE scope at org.apache.solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest:
>    1) Thread[id=9359, name=Connection evictor, state=TIMED_WAITING, 
> group=TGRP-HdfsChaosMonkeyNothingIsSafeTest]
>         at java.lang.Thread.sleep(Native Method)
>         at 
> org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
>         at java.lang.Thread.run(Thread.java:748)
>    2) Thread[id=9360, 
> name=TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[EE9E6F4F6108C35B]-SendThread(127.0.0.1:39603),
>  state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeyNothingIsSafeTest]
>         at java.lang.Thread.sleep(Native Method)
>         at 
> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)
>         at 
> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)
>         at 
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)
>    3) Thread[id=9361, 
> name=TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[EE9E6F4F6108C35B]-EventThread,
>  state=WAITING, group=TGRP-HdfsChaosMonkeyNothingIsSafeTest]
>         at sun.misc.Unsafe.park(Native Method)
>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>         at 
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at 
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
>    4) Thread[id=9362, name=zkConnectionManagerCallback-4269-thread-1, 
> state=WAITING, group=TGRP-HdfsChaosMonkeyNothingIsSafeTest]
>         at sun.misc.Unsafe.park(Native Method)
>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>         at 
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
>         at __randomizedtesting.SeedInfo.seed([EE9E6F4F6108C35B]:0)
>
>
> FAILED:  
> junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest
>
> Error Message:
> There are still zombie threads that couldn't be terminated:    1) 
> Thread[id=9360, 
> name=TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[EE9E6F4F6108C35B]-SendThread(127.0.0.1:39603),
>  state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeyNothingIsSafeTest]         at 
> java.lang.Thread.sleep(Native Method)         at 
> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)
>          at 
> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000) 
>         at 
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)    2) 
> Thread[id=9362, name=zkConnectionManagerCallback-4269-thread-1, 
> state=WAITING, group=TGRP-HdfsChaosMonkeyNothingIsSafeTest]         at 
> sun.misc.Unsafe.park(Native Method)         at 
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>          at 
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)   
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) 
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>          at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>          at java.lang.Thread.run(Thread.java:748)
>
> Stack Trace:
> com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie 
> threads that couldn't be terminated:
>    1) Thread[id=9360, 
> name=TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[EE9E6F4F6108C35B]-SendThread(127.0.0.1:39603),
>  state=TIMED_WAITING, group=TGRP-HdfsChaosMonkeyNothingIsSafeTest]
>         at java.lang.Thread.sleep(Native Method)
>         at 
> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)
>         at 
> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)
>         at 
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)
>    2) Thread[id=9362, name=zkConnectionManagerCallback-4269-thread-1, 
> state=WAITING, group=TGRP-HdfsChaosMonkeyNothingIsSafeTest]
>         at sun.misc.Unsafe.park(Native Method)
>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>         at 
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
>         at __randomizedtesting.SeedInfo.seed([EE9E6F4F6108C35B]:0)
>
>
> FAILED:  org.apache.solr.cloud.hdfs.StressHdfsTest.test
>
> Error Message:
> Timeout occured while waiting response from server at: 
> http://127.0.0.1:33555/_o/uz
>
> Stack Trace:
> org.apache.solr.client.solrj.SolrServerException: Timeout occured while 
> waiting response from server at: http://127.0.0.1:33555/_o/uz
>         at 
> __randomizedtesting.SeedInfo.seed([EE9E6F4F6108C35B:66CA5095CFF4AEA3]:0)
>         at 
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
>         at 
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
>         at 
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
>         at 
> org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
>         at 
> org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
>         at 
> org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1106)
>         at 
> org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:886)
>         at 
> org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:819)
>         at 
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
>         at 
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
>         at 
> org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:425)
>         at 
> org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:341)
>         at 
> org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1006)
>         at 
> org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:983)
>         at 
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>         at 
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
>         at 
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>         at 
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
>         at 
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>         at 
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>         at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at 
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>         at 
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
>         at 
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
>         at 
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
>         at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
>         at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
>         at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
>         at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at 
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>         at 
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>         at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at 
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>         at 
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>         at 
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>         at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at 
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>         at 
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>         at 
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>         at 
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
>         at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at 
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>         at java.lang.Thread.run(Thread.java:748)
> Caused by: java.net.SocketTimeoutException: Read timed out
>         at java.net.SocketInputStream.socketRead0(Native Method)
>         at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
>         at java.net.SocketInputStream.read(SocketInputStream.java:171)
>         at java.net.SocketInputStream.read(SocketInputStream.java:141)
>         at 
> org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
>         at 
> org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
>         at 
> org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
>         at 
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
>         at 
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
>         at 
> org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
>         at 
> org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
>         at 
> org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
>         at 
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
>         at 
> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
>         at 
> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
>         at 
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
>         at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
>         at 
> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
>         at 
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
>         at 
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
>         at 
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
>         at 
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
>         ... 44 more
>
>
> FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.hdfs.StressHdfsTest
>
> Error Message:
> 158 threads leaked from SUITE scope at 
> org.apache.solr.cloud.hdfs.StressHdfsTest:     1) Thread[id=8104, 
> name=TEST-StressHdfsTest.test-seed#[EE9E6F4F6108C35B]-SendThread(127.0.0.1:33783),
>  state=TIMED_WAITING, group=TGRP-StressHdfsTest]         at 
> java.lang.Thread.sleep(Native Method)         at 
> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)
>          at 
> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000) 
>         at 
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)    2) 
> Thread[id=8333, name=zkConnectionManagerCallback-2254-thread-1-EventThread, 
> state=WAITING, group=TGRP-StressHdfsTest]         at 
> sun.misc.Unsafe.park(Native Method)         at 
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>          at 
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)   
>       at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502) 
>    3) Thread[id=8049, name=Connection evictor, state=TIMED_WAITING, 
> group=TGRP-StressHdfsTest]         at java.lang.Thread.sleep(Native Method)   
>       at 
> org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
>          at java.lang.Thread.run(Thread.java:748)    4) Thread[id=8060, 
> name=zkConnectionManagerCallback-2237-thread-1, state=WAITING, 
> group=TGRP-StressHdfsTest]         at sun.misc.Unsafe.park(Native Method)     
>     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)      
>    at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>          at 
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)   
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) 
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>          at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>          at java.lang.Thread.run(Thread.java:748)    5) Thread[id=8038, 
> name=qtp859078352-8038, state=TIMED_WAITING, group=TGRP-StressHdfsTest]       
>   at sun.misc.Unsafe.park(Native Method)         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)        
>  at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>          at 
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)   
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
>          at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
>          at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
>          at java.lang.Thread.run(Thread.java:748)    6) Thread[id=8260, 
> name=Connection evictor, state=TIMED_WAITING, group=TGRP-StressHdfsTest]      
>    at java.lang.Thread.sleep(Native Method)         at 
> org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
>          at java.lang.Thread.run(Thread.java:748)    7) Thread[id=8098, 
> name=Connection evictor, state=TIMED_WAITING, group=TGRP-StressHdfsTest]      
>    at java.lang.Thread.sleep(Native Method)         at 
> org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
>          at java.lang.Thread.run(Thread.java:748)    8) Thread[id=8284, 
> name=SolrRrdBackendFactory-3061-thread-2, state=WAITING, 
> group=TGRP-StressHdfsTest]         at sun.misc.Unsafe.park(Native Method)     
>     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)      
>    at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>          at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
>          at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>          at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) 
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>          at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>          at java.lang.Thread.run(Thread.java:748)    9) Thread[id=8012, 
> name=Connection evictor, state=TIMED_WAITING, group=TGRP-StressHdfsTest]      
>    at java.lang.Thread.sleep(Native Method)         at 
> org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
>          at java.lang.Thread.run(Thread.java:748)   10) Thread[id=8127, 
> name=searcherExecutor-3064-thread-1, state=WAITING, 
> group=TGRP-StressHdfsTest]         at sun.misc.Unsafe.park(Native Method)     
>     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)      
>    at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>          at 
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)   
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) 
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>          at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>          at java.lang.Thread.run(Thread.java:748)   11) Thread[id=8036, 
> name=qtp859078352-8036, state=RUNNABLE, group=TGRP-StressHdfsTest]         at 
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at 
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at 
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at 
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at 
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at 
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at 
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
>          at 
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
>          at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
>          at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
>          at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
>          at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
>          at 
> org.eclipse.jetty.io.ManagedSelector$$Lambda$30/1990726269.run(Unknown 
> Source)         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
>          at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
>          at java.lang.Thread.run(Thread.java:748)   12) Thread[id=8166, 
> name=MetricsHistoryHandler-3076-thread-1, state=TIMED_WAITING, 
> group=TGRP-StressHdfsTest]         at sun.misc.Unsafe.park(Native Method)     
>     at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) 
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>          at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>          at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>          at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) 
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>          at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>          at java.lang.Thread.run(Thread.java:748)   13) Thread[id=8138, 
> name=qtp1864807748-8138, state=RUNNABLE, group=TGRP-StressHdfsTest]         
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at 
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at 
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at 
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at 
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at 
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at 
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
>          at 
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
>          at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
>          at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
>          at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
>          at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
>          at 
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
>          at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
>          at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
>          at java.lang.Thread.run(Thread.java:748)   14) Thread[id=8261, 
> name=Connection evictor, state=TIMED_WAITING, group=TGRP-StressHdfsTest]      
>    at java.lang.Thread.sleep(Native Method)         at 
> org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
>          at java.lang.Thread.run(Thread.java:748)   15) Thread[id=8197, 
> name=qtp2124653598-8197, state=TIMED_WAITING, group=TGRP-StressHdfsTest]      
>    at sun.misc.Unsafe.park(Native Method)         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)        
>  at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>          at 
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)   
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
>          at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
>          at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
>          at java.lang.Thread.run(Thread.java:748)   16) Thread[id=8220, 
> name=SolrRrdBackendFactory-3089-thread-1, state=TIMED_WAITING, 
> group=TGRP-StressHdfsTest]         at sun.misc.Unsafe.park(Native Method)     
>     at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) 
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>          at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>          at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>          at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) 
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>          at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>          at java.lang.Thread.run(Thread.java:748)   17) Thread[id=8157, 
> name=TEST-StressHdfsTest.test-seed#[EE9E6F4F6108C35B]-EventThread, 
> state=WAITING, group=TGRP-StressHdfsTest]         at 
> sun.misc.Unsafe.park(Native Method)         at 
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>          at 
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)   
>       at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502) 
>   18) Thread[id=8251, name=qtp1464698720-8251, state=TIMED_WAITING, 
> group=TGRP-StressHdfsTest]         at sun.misc.Unsafe.park(Native Method)     
>     at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) 
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>          at 
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)   
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
>          at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
>          at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
>          at java.lang.Thread.run(Thread.java:748)   19) Thread[id=8055, 
> name=zkConnectionManagerCallback-2230-thread-1, state=WAITING, 
> group=TGRP-StressHdfsTest]         at sun.misc.Unsafe.park(Native Method)     
>     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)      
>    at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>          at 
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)   
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) 
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>          at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>          at java.lang.Thread.run(Thread.java:748)   20) Thread[id=8057, 
> name=Connection evictor, state=TIMED_WAITING, group=TGRP-StressHdfsTest]      
>    at java.lang.Thread.sleep(Native Method)         at 
> org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
>          at java.lang.Thread.run(Thread.java:748)   21) Thread[id=8002, 
> name=Connection evictor, state=TIMED_WAITING, group=TGRP-StressHdfsTest]      
>    at java.lang.Thread.sleep(Native Method)         at 
> org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
>          at java.lang.Thread.run(Thread.java:748)   22) Thread[id=8199, 
> name=Scheduler-353609091, state=TIMED_WAITING, group=TGRP-StressHdfsTest]     
>     at sun.misc.Unsafe.park(Native Method)         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)        
>  at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>          at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>          at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>          at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) 
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>          at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>          at java.lang.Thread.run(Thread.java:748)   23) Thread[id=8282, 
> name=SolrRrdBackendFactory-3033-thread-2, state=WAITING, 
> group=TGRP-StressHdfsTest]         at sun.misc.Unsafe.park(Native Method)     
>     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)      
>    at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>          at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
>          at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>          at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) 
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>          at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>          at java.lang.Thread.run(Thread.java:748)   24) Thread[id=8191, 
> name=qtp2124653598-8191, state=TIMED_WAITING, group=TGRP-StressHdfsTest]      
>    at sun.misc.Unsafe.park(Native Method)         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)        
>  at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>          at 
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)   
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
>          at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
>          at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
>          at java.lang.Thread.run(Thread.java:748)   25) Thread[id=8017, 
> name=MetricsHistoryHandler-3034-thread-1, state=TIMED_WAITING, 
> group=TGRP-StressHdfsTest]         at sun.misc.Unsafe.park(Native Method)     
>     at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) 
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>          at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>          at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>          at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) 
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>          at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>          at java.lang.Thread.run(Thread.java:748)   26) Thread[id=8143, 
> name=qtp1864807748-8143, state=TIMED_WAITING, group=TGRP-StressHdfsTest]      
>    at sun.misc.Unsafe.park(Native Method)         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)        
>  at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>          at 
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)   
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
>          at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
>          at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
>          at java.lang.Thread.run(Thread.java:748)   27) Thread[id=8092, 
> name=qtp1773868801-8092, state=TIMED_WAITING, group=TGRP-StressHdfsTest]      
>    at sun.misc.Unsafe.park(Native Method)         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)        
>  at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
>          at 
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:292)
>          at 
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:357)
>          at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
>          at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
>          at java.lang.Thread.run(Thread.java:748)   28) Thread[id=8030, 
> name=searcherExecutor-3036-thread-1, state=WAITING, 
> group=TGRP-StressHdfsTest]         at sun.misc.Unsafe.park(Native Method)     
>     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)      
>    at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>          at 
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)   
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) 
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>          at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>          at java.lang.Thread.run(Thread.java:748)   29) Thread[id=8225, 
> name=Scheduler-891485727, state=WAITING, group=TGRP-StressHdfsTest]         
> at sun.misc.Unsafe.park(Native Method)         at 
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>          at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
>          at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>          at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) 
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>          at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>          at java.lang.Thread.run(Thread.java:748)   30) Thread[id=8114, 
> name=Thread-1644, state=WAITING, group=TGRP-StressHdfsTest]         at 
> java.lang.Object.wait(Native Method)         at 
> java.lang.Object.wait(Object.java:502)         at 
> org.apache.solr.core.CloserThread.run(CoreContainer.java:1841)   31) 
> Thread[id=8004, name=Connection evictor, state=TIMED_WAITING, 
> group=TGRP-StressHdfsTest]         at java.lang.Thread.sleep(Native Method)   
>       at 
> org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
>          at java.lang.Thread.run(Thread.java:748)   32) Thread[id=8331, 
> name=zkConnectionManagerCallback-2221-thread-1-SendThread(127.0.0.1:33783), 
> state=TIMED_WAITING, group=TGRP-StressHdfsTest]         at 
> java.lang.Thread.sleep(Native Method)         at 
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.ja

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Reply via email to