Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-7.x/62/

7 tests failed.
FAILED:  org.apache.solr.cloud.TestAuthenticationFramework.testBasics

Error Message:
Error from server at 
http://127.0.0.1:41715/solr/testcollection_shard1_replica_n3: Expected mime 
type application/octet-stream but got text/html. <html> <head> <meta 
http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 404 
Can not find: /solr/testcollection_shard1_replica_n3/update</title> </head> 
<body><h2>HTTP ERROR 404</h2> <p>Problem accessing 
/solr/testcollection_shard1_replica_n3/update. Reason: <pre>    Can not find: 
/solr/testcollection_shard1_replica_n3/update</pre></p><hr><a 
href="http://eclipse.org/jetty";>Powered by Jetty:// 9.4.10.v20180503</a><hr/>  
</body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from 
server at http://127.0.0.1:41715/solr/testcollection_shard1_replica_n3: 
Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Can not find: 
/solr/testcollection_shard1_replica_n3/update</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/testcollection_shard1_replica_n3/update. Reason:
<pre>    Can not find: 
/solr/testcollection_shard1_replica_n3/update</pre></p><hr><a 
href="http://eclipse.org/jetty";>Powered by Jetty:// 9.4.10.v20180503</a><hr/>

</body>
</html>

        at 
__randomizedtesting.SeedInfo.seed([3954E34127A9E6C4:48C4D6D1F47B8B4]:0)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:551)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1015)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:886)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:948)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:948)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:948)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:948)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:948)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:819)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at 
org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:233)
        at 
org.apache.solr.cloud.TestAuthenticationFramework.collectionCreateSearchDeleteTwice(TestAuthenticationFramework.java:127)
        at 
org.apache.solr.cloud.TestAuthenticationFramework.testBasics(TestAuthenticationFramework.java:75)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: 
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at http://127.0.0.1:41715/solr/testcollection_shard1_replica_n3: 
Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Can not find: 
/solr/testcollection_shard1_replica_n3/update</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/testcollection_shard1_replica_n3/update. Reason:
<pre>    Can not find: 
/solr/testcollection_shard1_replica_n3/update</pre></p><hr><a 
href="http://eclipse.org/jetty";>Powered by Jetty:// 9.4.10.v20180503</a><hr/>

</body>
</html>

        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:528)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:192)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        ... 1 more


FAILED:  
org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest.testSimple

Error Message:
Waiting for collection testSimple2 null Live Nodes: [127.0.0.1:41379_solr, 
127.0.0.1:44234_solr] Last available state: 
DocCollection(testSimple2//collections/testSimple2/state.json/23)={   
"pullReplicas":"0",   "replicationFactor":"2",   "shards":{     "shard1":{      
 "range":"80000000-ffffffff",       "state":"active",       "replicas":{        
 "core_node2":{           
"dataDir":"hdfs://localhost:57014/data/testSimple2/core_node2/data/",           
"base_url":"http://127.0.0.1:44234/solr";,           
"node_name":"127.0.0.1:44234_solr",           "type":"NRT",           
"force_set_state":"false",           
"ulogDir":"hdfs://localhost:57014/data/testSimple2/core_node2/data/tlog",       
    "core":"testSimple2_shard1_replica_n1",           "shared_storage":"true",  
         "state":"active",           "leader":"true"},         "core_node4":{   
        "dataDir":"hdfs://localhost:57014/data/testSimple2/core_node4/data/",   
        "base_url":"http://127.0.0.1:52293/solr";,           
"node_name":"127.0.0.1:52293_solr",           "type":"NRT",           
"force_set_state":"false",           
"ulogDir":"hdfs://localhost:57014/data/testSimple2/core_node4/data/tlog",       
    "core":"testSimple2_shard1_replica_n3",           "shared_storage":"true",  
         "state":"down"}}},     "shard2":{       "range":"0-7fffffff",       
"state":"active",       "replicas":{         "core_node6":{           
"dataDir":"hdfs://localhost:57014/data/testSimple2/core_node6/data/",           
"base_url":"http://127.0.0.1:44234/solr";,           
"node_name":"127.0.0.1:44234_solr",           "type":"NRT",           
"force_set_state":"false",           
"ulogDir":"hdfs://localhost:57014/data/testSimple2/core_node6/data/tlog",       
    "core":"testSimple2_shard2_replica_n5",           "shared_storage":"true",  
         "state":"active",           "leader":"true"},         "core_node8":{   
        "dataDir":"hdfs://localhost:57014/data/testSimple2/core_node8/data/",   
        "base_url":"http://127.0.0.1:52293/solr";,           
"node_name":"127.0.0.1:52293_solr",           "type":"NRT",           
"force_set_state":"false",           
"ulogDir":"hdfs://localhost:57014/data/testSimple2/core_node8/data/tlog",       
    "core":"testSimple2_shard2_replica_n7",           "shared_storage":"true",  
         "state":"down"}}}},   "router":{"name":"compositeId"},   
"maxShardsPerNode":"2",   "autoAddReplicas":"true",   "nrtReplicas":"2",   
"tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Waiting for collection testSimple2
null
Live Nodes: [127.0.0.1:41379_solr, 127.0.0.1:44234_solr]
Last available state: 
DocCollection(testSimple2//collections/testSimple2/state.json/23)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{
        "core_node2":{
          "dataDir":"hdfs://localhost:57014/data/testSimple2/core_node2/data/",
          "base_url":"http://127.0.0.1:44234/solr";,
          "node_name":"127.0.0.1:44234_solr",
          "type":"NRT",
          "force_set_state":"false",
          
"ulogDir":"hdfs://localhost:57014/data/testSimple2/core_node2/data/tlog",
          "core":"testSimple2_shard1_replica_n1",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"},
        "core_node4":{
          "dataDir":"hdfs://localhost:57014/data/testSimple2/core_node4/data/",
          "base_url":"http://127.0.0.1:52293/solr";,
          "node_name":"127.0.0.1:52293_solr",
          "type":"NRT",
          "force_set_state":"false",
          
"ulogDir":"hdfs://localhost:57014/data/testSimple2/core_node4/data/tlog",
          "core":"testSimple2_shard1_replica_n3",
          "shared_storage":"true",
          "state":"down"}}},
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{
        "core_node6":{
          "dataDir":"hdfs://localhost:57014/data/testSimple2/core_node6/data/",
          "base_url":"http://127.0.0.1:44234/solr";,
          "node_name":"127.0.0.1:44234_solr",
          "type":"NRT",
          "force_set_state":"false",
          
"ulogDir":"hdfs://localhost:57014/data/testSimple2/core_node6/data/tlog",
          "core":"testSimple2_shard2_replica_n5",
          "shared_storage":"true",
          "state":"active",
          "leader":"true"},
        "core_node8":{
          "dataDir":"hdfs://localhost:57014/data/testSimple2/core_node8/data/",
          "base_url":"http://127.0.0.1:52293/solr";,
          "node_name":"127.0.0.1:52293_solr",
          "type":"NRT",
          "force_set_state":"false",
          
"ulogDir":"hdfs://localhost:57014/data/testSimple2/core_node8/data/tlog",
          "core":"testSimple2_shard2_replica_n7",
          "shared_storage":"true",
          "state":"down"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"true",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
        at 
__randomizedtesting.SeedInfo.seed([3954E34127A9E6C4:1E7C7BF005A3215]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at 
org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:278)
        at 
org.apache.solr.cloud.autoscaling.AutoAddReplicasIntegrationTest.testSimple(AutoAddReplicasIntegrationTest.java:129)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.cloud.autoscaling.sim.TestGenericDistributedQueue.testDistributedQueue

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([3954E34127A9E6C4]:0)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.autoscaling.sim.TestGenericDistributedQueue

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([3954E34127A9E6C4]:0)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testBasic

Error Message:
there should be new MOVERPLICA ops

Stack Trace:
java.lang.AssertionError: there should be new MOVERPLICA ops
        at 
__randomizedtesting.SeedInfo.seed([3954E34127A9E6C4:92AEFE54F87560EA]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at 
org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testBasic(TestLargeCluster.java:225)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testNodeLost

Error Message:
last state: DocCollection(testNodeLost//clusterstate.json/93)={   
"replicationFactor":"10",   "pullReplicas":"0",   
"router":{"name":"compositeId"},   "maxShardsPerNode":"5",   
"autoAddReplicas":"false",   "nrtReplicas":"10",   "tlogReplicas":"0",   
"autoCreated":"true",   "shards":{     "shard20":{       "replicas":{         
"core_node191":{           "core":"testNodeLost_shard20_replica_n191",          
 "leader":"true",           "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10011_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         
"core_node258":{           "node_name":"127.0.0.1:10009_solr",           
"core":"testNodeLost_shard20_replica_n257",           "state":"active",         
  "INDEX.sizeInBytes":1,           "type":"NRT"},         "core_node200":{      
     "core":"testNodeLost_shard20_replica_n200",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10028_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node266":{           
"node_name":"127.0.0.1:10028_solr",           
"core":"testNodeLost_shard20_replica_n265",           "state":"active",         
  "INDEX.sizeInBytes":1,           "type":"NRT"},         "core_node199":{      
     "core":"testNodeLost_shard20_replica_n199",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10074_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node196":{           
"core":"testNodeLost_shard20_replica_n196",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10071_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node195":{           
"core":"testNodeLost_shard20_replica_n195",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10092_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node250":{           
"node_name":"127.0.0.1:10011_solr",           
"core":"testNodeLost_shard20_replica_n249",           "state":"active",         
  "INDEX.sizeInBytes":1,           "type":"NRT"},         "core_node193":{      
     "core":"testNodeLost_shard20_replica_n193",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10079_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node192":{           
"core":"testNodeLost_shard20_replica_n192",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10091_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"73330000-7fffffff",       
"state":"active"},     "shard10":{       "replicas":{         "core_node92":{   
        "core":"testNodeLost_shard10_replica_n92",           "leader":"true",   
        "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10091_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node91":{ 
          "core":"testNodeLost_shard10_replica_n91",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10011_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node93":{           
"core":"testNodeLost_shard10_replica_n93",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10079_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node268":{           
"node_name":"127.0.0.1:10044_solr",           
"core":"testNodeLost_shard10_replica_n267",           "state":"active",         
  "INDEX.sizeInBytes":1,           "type":"NRT"},         "core_node96":{       
    "core":"testNodeLost_shard10_replica_n96",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10071_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node95":{           
"core":"testNodeLost_shard10_replica_n95",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10092_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node100":{           
"core":"testNodeLost_shard10_replica_n100",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10028_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node252":{           
"node_name":"127.0.0.1:10014_solr",           
"core":"testNodeLost_shard10_replica_n251",           "state":"active",         
  "INDEX.sizeInBytes":1,           "type":"NRT"},         "core_node99":{       
    "core":"testNodeLost_shard10_replica_n99",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10074_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node260":{           
"node_name":"127.0.0.1:10087_solr",           
"core":"testNodeLost_shard10_replica_n259",           "state":"active",         
  "INDEX.sizeInBytes":1,           "type":"NRT"}},       
"range":"f3330000-ffffffff",       "state":"active"},     "shard11":{       
"replicas":{         "core_node108":{           
"core":"testNodeLost_shard11_replica_n108",           "leader":"true",          
 "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,     
      "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10056_solr",      
     "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node107":{           
"core":"testNodeLost_shard11_replica_n107",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10002_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node105":{           
"core":"testNodeLost_shard11_replica_n105",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10070_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node104":{           
"core":"testNodeLost_shard11_replica_n104",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10098_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node236":{           
"node_name":"127.0.0.1:10069_solr",           
"core":"testNodeLost_shard11_replica_n235",           "state":"active",         
  "INDEX.sizeInBytes":1,           "type":"NRT"},         "core_node103":{      
     "core":"testNodeLost_shard11_replica_n103",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10057_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node102":{           
"core":"testNodeLost_shard11_replica_n102",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10094_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node101":{           
"core":"testNodeLost_shard11_replica_n101",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10060_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node110":{           
"core":"testNodeLost_shard11_replica_n110",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10010_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node109":{           
"core":"testNodeLost_shard11_replica_n109",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10034_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"0-ccbffff",       
"state":"active"},     "shard12":{       "replicas":{         "core_node119":{  
         "core":"testNodeLost_shard12_replica_n119",           "leader":"true", 
          "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10032_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         
"core_node118":{           "core":"testNodeLost_shard12_replica_n118",          
 "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,     
      "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10083_solr",      
     "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node216":{           
"node_name":"127.0.0.1:10034_solr",           
"core":"testNodeLost_shard12_replica_n215",           "state":"active",         
  "INDEX.sizeInBytes":1,           "type":"NRT"},         "core_node115":{      
     "core":"testNodeLost_shard12_replica_n115",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10031_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node114":{           
"core":"testNodeLost_shard12_replica_n114",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10014_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node113":{           
"core":"testNodeLost_shard12_replica_n113",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10044_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node212":{           
"node_name":"127.0.0.1:10094_solr",           
"core":"testNodeLost_shard12_replica_n211",           "state":"active",         
  "INDEX.sizeInBytes":1,           "type":"NRT"},         "core_node112":{      
     "core":"testNodeLost_shard12_replica_n112",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10016_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node120":{           
"core":"testNodeLost_shard12_replica_n120",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10046_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node208":{           
"node_name":"127.0.0.1:10053_solr",           
"core":"testNodeLost_shard12_replica_n207",           "state":"active",         
  "INDEX.sizeInBytes":1,           "type":"NRT"}},       
"range":"ccc0000-1998ffff",       "state":"active"},     "shard2":{       
"replicas":{         "core_node18":{           
"core":"testNodeLost_shard2_replica_n18",           "leader":"true",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10083_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node206":{           
"node_name":"127.0.0.1:10059_solr",           
"core":"testNodeLost_shard2_replica_n205",           "state":"active",          
 "INDEX.sizeInBytes":1,           "type":"NRT"},         "core_node19":{        
   "core":"testNodeLost_shard2_replica_n19",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10032_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node214":{           
"node_name":"127.0.0.1:10091_solr",           
"core":"testNodeLost_shard2_replica_n213",           "state":"active",          
 "INDEX.sizeInBytes":1,           "type":"NRT"},         "core_node210":{       
    "node_name":"127.0.0.1:10037_solr",           
"core":"testNodeLost_shard2_replica_n209",           "state":"active",          
 "INDEX.sizeInBytes":1,           "type":"NRT"},         "core_node20":{        
   "core":"testNodeLost_shard2_replica_n20",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10046_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node12":{           
"core":"testNodeLost_shard2_replica_n12",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10016_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node14":{           
"core":"testNodeLost_shard2_replica_n14",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10014_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node13":{           
"core":"testNodeLost_shard2_replica_n13",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10044_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node15":{           
"core":"testNodeLost_shard2_replica_n15",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10031_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"8ccc0000-9998ffff",       
"state":"active"},     "shard13":{       "replicas":{         "core_node129":{  
         "core":"testNodeLost_shard13_replica_n129",           "leader":"true", 
          "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10077_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         
"core_node128":{           "core":"testNodeLost_shard13_replica_n128",          
 "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,     
      "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10084_solr",      
     "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node127":{           
"core":"testNodeLost_shard13_replica_n127",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10050_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node126":{           
"core":"testNodeLost_shard13_replica_n126",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10009_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node125":{           
"core":"testNodeLost_shard13_replica_n125",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10045_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node124":{           
"core":"testNodeLost_shard13_replica_n124",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10069_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node123":{           
"core":"testNodeLost_shard13_replica_n123",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10023_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node122":{           
"core":"testNodeLost_shard13_replica_n122",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10041_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node121":{           
"core":"testNodeLost_shard13_replica_n121",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10099_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node130":{           
"core":"testNodeLost_shard13_replica_n130",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10073_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"19990000-2665ffff",       
"state":"active"},     "shard14":{       "replicas":{         "core_node139":{  
         "core":"testNodeLost_shard14_replica_n139",           "leader":"true", 
          "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10093_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         
"core_node138":{           "core":"testNodeLost_shard14_replica_n138",          
 "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,     
      "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10063_solr",      
     "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node137":{           
"core":"testNodeLost_shard14_replica_n137",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10055_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node135":{           
"core":"testNodeLost_shard14_replica_n135",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10068_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node134":{           
"core":"testNodeLost_shard14_replica_n134",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10075_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node133":{           
"core":"testNodeLost_shard14_replica_n133",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10026_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node132":{           
"core":"testNodeLost_shard14_replica_n132",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10038_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node131":{           
"core":"testNodeLost_shard14_replica_n131",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10052_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node262":{           
"node_name":"127.0.0.1:10041_solr",           
"core":"testNodeLost_shard14_replica_n261",           "state":"active",         
  "INDEX.sizeInBytes":1,           "type":"NRT"},         "core_node140":{      
     "core":"testNodeLost_shard14_replica_n140",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10037_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"26660000-3332ffff",       
"state":"active"},     "shard3":{       "replicas":{         "core_node29":{    
       "core":"testNodeLost_shard3_replica_n29",           "leader":"true",     
      "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10077_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node28":{ 
          "core":"testNodeLost_shard3_replica_n28",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10084_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node30":{           
"core":"testNodeLost_shard3_replica_n30",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10073_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node21":{           
"core":"testNodeLost_shard3_replica_n21",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10099_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node23":{           
"core":"testNodeLost_shard3_replica_n23",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10023_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node22":{           
"core":"testNodeLost_shard3_replica_n22",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10041_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node25":{           
"core":"testNodeLost_shard3_replica_n25",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10045_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node24":{           
"core":"testNodeLost_shard3_replica_n24",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10069_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node27":{           
"core":"testNodeLost_shard3_replica_n27",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10050_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node26":{           
"core":"testNodeLost_shard3_replica_n26",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10009_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"99990000-a665ffff",       
"state":"active"},     "shard15":{       "replicas":{         "core_node149":{  
         "core":"testNodeLost_shard15_replica_n149",           "leader":"true", 
          "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10029_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         
"core_node147":{           "core":"testNodeLost_shard15_replica_n147",          
 "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,     
      "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10086_solr",      
     "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node246":{           
"node_name":"127.0.0.1:10085_solr",           
"core":"testNodeLost_shard15_replica_n245",           "state":"active",         
  "INDEX.sizeInBytes":1,           "type":"NRT"},         "core_node146":{      
     "core":"testNodeLost_shard15_replica_n146",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10061_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node145":{           
"core":"testNodeLost_shard15_replica_n145",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10024_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node144":{           
"core":"testNodeLost_shard15_replica_n144",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10001_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node143":{           
"core":"testNodeLost_shard15_replica_n143",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10027_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node142":{           
"core":"testNodeLost_shard15_replica_n142",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10089_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node141":{           
"core":"testNodeLost_shard15_replica_n141",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10039_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node150":{           
"core":"testNodeLost_shard15_replica_n150",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10076_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"33330000-3fffffff",       
"state":"active"},     "shard4":{       "replicas":{         "core_node39":{    
       "core":"testNodeLost_shard4_replica_n39",           "leader":"true",     
      "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10093_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node40":{ 
          "core":"testNodeLost_shard4_replica_n40",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10037_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node32":{           
"core":"testNodeLost_shard4_replica_n32",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10038_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node31":{           
"core":"testNodeLost_shard4_replica_n31",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10052_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node264":{           
"node_name":"127.0.0.1:10092_solr",           
"core":"testNodeLost_shard4_replica_n263",           "state":"active",          
 "INDEX.sizeInBytes":1,           "type":"NRT"},         "core_node34":{        
   "core":"testNodeLost_shard4_replica_n34",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10075_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node33":{           
"core":"testNodeLost_shard4_replica_n33",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10026_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node35":{           
"core":"testNodeLost_shard4_replica_n35",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10068_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node38":{           
"core":"testNodeLost_shard4_replica_n38",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10063_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node37":{           
"core":"testNodeLost_shard4_replica_n37",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10055_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"a6660000-b332ffff",       
"state":"active"},     "shard16":{       "replicas":{         "core_node218":{  
         "node_name":"127.0.0.1:10031_solr",           
"core":"testNodeLost_shard16_replica_n217",           "state":"active",         
  "INDEX.sizeInBytes":1,           "type":"NRT"},         "core_node238":{      
     "node_name":"127.0.0.1:10015_solr",           
"core":"testNodeLost_shard16_replica_n237",           "state":"active",         
  "INDEX.sizeInBytes":1,           "type":"NRT"},         "core_node159":{      
     "core":"testNodeLost_shard16_replica_n159",           "leader":"true",     
      "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10064_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         
"core_node202":{           "node_name":"127.0.0.1:10072_solr",           
"core":"testNodeLost_shard16_replica_n201",           "state":"active",         
  "INDEX.sizeInBytes":1,           "type":"NRT"},         "core_node156":{      
     "core":"testNodeLost_shard16_replica_n156",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10062_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node155":{           
"core":"testNodeLost_shard16_replica_n155",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10048_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node242":{           
"node_name":"127.0.0.1:10068_solr",           
"core":"testNodeLost_shard16_replica_n241",           "state":"active",         
  "INDEX.sizeInBytes":1,           "type":"NRT"},         "core_node153":{      
     "core":"testNodeLost_shard16_replica_n153",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10085_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node151":{           
"core":"testNodeLost_shard16_replica_n151",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10012_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node160":{           
"core":"testNodeLost_shard16_replica_n160",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10053_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"40000000-4ccbffff",       
"state":"active"},     "shard5":{       "replicas":{         "core_node248":{   
        "node_name":"127.0.0.1:10040_solr",           
"core":"testNodeLost_shard5_replica_n247",           "state":"active",          
 "INDEX.sizeInBytes":1,           "type":"NRT"},         "core_node50":{        
   "core":"testNodeLost_shard5_replica_n50",           "leader":"true",         
  "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,    
       "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10076_solr",     
      "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node41":{           
"core":"testNodeLost_shard5_replica_n41",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10039_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node43":{           
"core":"testNodeLost_shard5_replica_n43",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10027_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node42":{           
"core":"testNodeLost_shard5_replica_n42",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10089_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node45":{           
"core":"testNodeLost_shard5_replica_n45",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10024_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node44":{           
"core":"testNodeLost_shard5_replica_n44",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10001_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node47":{           
"core":"testNodeLost_shard5_replica_n47",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10086_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node46":{           
"core":"testNodeLost_shard5_replica_n46",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10061_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node49":{           
"core":"testNodeLost_shard5_replica_n49",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10029_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"b3330000-bfffffff",       
"state":"active"},     "shard17":{       "replicas":{         "core_node169":{  
         "core":"testNodeLost_shard17_replica_n169",           "leader":"true", 
          "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10040_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         
"core_node168":{           "core":"testNodeLost_shard17_replica_n168",          
 "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,     
      "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10067_solr",      
     "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node167":{           
"core":"testNodeLost_shard17_replica_n167",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10058_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node222":{           
"node_name":"127.0.0.1:10002_solr",           
"core":"testNodeLost_shard17_replica_n221",           "state":"active",         
  "INDEX.sizeInBytes":1,           "type":"NRT"},         "core_node165":{      
     "core":"testNodeLost_shard17_replica_n165",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10008_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node164":{           
"core":"testNodeLost_shard17_replica_n164",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10072_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node163":{           
"core":"testNodeLost_shard17_replica_n163",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10087_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node162":{           
"core":"testNodeLost_shard17_replica_n162",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10097_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node161":{           
"core":"testNodeLost_shard17_replica_n161",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10007_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node170":{           
"core":"testNodeLost_shard17_replica_n170",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10036_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"4ccc0000-5998ffff",       
"state":"active"},     "shard18":{       "replicas":{         "core_node180":{  
         "core":"testNodeLost_shard18_replica_n180",           "leader":"true", 
          "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10013_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         
"core_node179":{           "core":"testNodeLost_shard18_replica_n179",          
 "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,     
      "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10043_solr",      
     "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node178":{           
"core":"testNodeLost_shard18_replica_n178",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10015_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node177":{           
"core":"testNodeLost_shard18_replica_n177",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10006_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node176":{           
"core":"testNodeLost_shard18_replica_n176",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10059_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node175":{           
"core":"testNodeLost_shard18_replica_n175",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10090_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node174":{           
"core":"testNodeLost_shard18_replica_n174",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10078_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node173":{           
"core":"testNodeLost_shard18_replica_n173",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10080_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node172":{           
"core":"testNodeLost_shard18_replica_n172",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10025_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node171":{           
"core":"testNodeLost_shard18_replica_n171",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10088_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"59990000-6665ffff",       
"state":"active"},     "shard19":{       "replicas":{         "core_node190":{  
         "core":"testNodeLost_shard19_replica_n190",           "leader":"true", 
          "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10000_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         
"core_node226":{           "node_name":"127.0.0.1:10075_solr",           
"core":"testNodeLost_shard19_replica_n225",           "state":"active",         
  "INDEX.sizeInBytes":1,           "type":"NRT"},         "core_node189":{      
     "core":"testNodeLost_shard19_replica_n189",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10042_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node188":{           
"core":"testNodeLost_shard19_replica_n188",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10003_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node254":{           
"node_name":"127.0.0.1:10030_solr",           
"core":"testNodeLost_shard19_replica_n253",           "state":"active",         
  "INDEX.sizeInBytes":1,           "type":"NRT"},         "core_node187":{      
     "core":"testNodeLost_shard19_replica_n187",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10054_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node230":{           
"node_name":"127.0.0.1:10024_solr",           
"core":"testNodeLost_shard19_replica_n229",           "state":"active",         
  "INDEX.sizeInBytes":1,           "type":"NRT"},         "core_node184":{      
     "core":"testNodeLost_shard19_replica_n184",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10095_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node183":{           
"core":"testNodeLost_shard19_replica_n183",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10081_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node181":{           
"core":"testNodeLost_shard19_replica_n181",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10030_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"66660000-7332ffff",       
"state":"active"},     "shard1":{       "replicas":{         "core_node1":{     
      "core":"testNodeLost_shard1_replica_n1",           "leader":"true",       
    "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,  
         "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10060_solr",   
        "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node234":{           
"node_name":"127.0.0.1:10062_solr",           
"core":"testNodeLost_shard1_replica_n233",           "state":"active",          
 "INDEX.sizeInBytes":1,           "type":"NRT"},         "core_node10":{        
   "core":"testNodeLost_shard1_replica_n10",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10010_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node7":{           
"core":"testNodeLost_shard1_replica_n7",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10002_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node8":{           
"core":"testNodeLost_shard1_replica_n8",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10056_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node9":{           
"core":"testNodeLost_shard1_replica_n9",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10034_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node2":{           
"core":"testNodeLost_shard1_replica_n2",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10094_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node3":{           
"core":"testNodeLost_shard1_replica_n3",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10057_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node4":{           
"core":"testNodeLost_shard1_replica_n4",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10098_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node5":{           
"core":"testNodeLost_shard1_replica_n5",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10070_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"80000000-8ccbffff",       
"state":"active"},     "shard6":{       "replicas":{         "core_node204":{   
        "node_name":"127.0.0.1:10056_solr",           
"core":"testNodeLost_shard6_replica_n203",           "state":"active",          
 "INDEX.sizeInBytes":1,           "type":"NRT"},         "core_node60":{        
   "core":"testNodeLost_shard6_replica_n60",           "leader":"true",         
  "SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,    
       "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10053_solr",     
      "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node51":{           
"core":"testNodeLost_shard6_replica_n51",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10012_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node244":{           
"node_name":"127.0.0.1:10046_solr",           
"core":"testNodeLost_shard6_replica_n243",           "state":"active",          
 "INDEX.sizeInBytes":1,           "type":"NRT"},         "core_node53":{        
   "core":"testNodeLost_shard6_replica_n53",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10085_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node220":{           
"node_name":"127.0.0.1:10097_solr",           
"core":"testNodeLost_shard6_replica_n219",           "state":"active",          
 "INDEX.sizeInBytes":1,           "type":"NRT"},         "core_node56":{        
   "core":"testNodeLost_shard6_replica_n56",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10062_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node55":{           
"core":"testNodeLost_shard6_replica_n55",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10048_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node240":{           
"node_name":"127.0.0.1:10063_solr",           
"core":"testNodeLost_shard6_replica_n239",           "state":"active",          
 "INDEX.sizeInBytes":1,           "type":"NRT"},         "core_node59":{        
   "core":"testNodeLost_shard6_replica_n59",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10064_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"c0000000-cccbffff",       
"state":"active"},     "shard7":{       "replicas":{         "core_node70":{    
       "core":"testNodeLost_shard7_replica_n70",           "leader":"true",     
      "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10036_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node61":{ 
          "core":"testNodeLost_shard7_replica_n61",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10007_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node224":{           
"node_name":"127.0.0.1:10078_solr",           
"core":"testNodeLost_shard7_replica_n223",           "state":"active",          
 "INDEX.sizeInBytes":1,           "type":"NRT"},         "core_node63":{        
   "core":"testNodeLost_shard7_replica_n63",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10087_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node62":{           
"core":"testNodeLost_shard7_replica_n62",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10097_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node65":{           
"core":"testNodeLost_shard7_replica_n65",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10008_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node64":{           
"core":"testNodeLost_shard7_replica_n64",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10072_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node67":{           
"core":"testNodeLost_shard7_replica_n67",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10058_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node69":{           
"core":"testNodeLost_shard7_replica_n69",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10040_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node68":{           
"core":"testNodeLost_shard7_replica_n68",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10067_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"cccc0000-d998ffff",       
"state":"active"},     "shard8":{       "replicas":{         "core_node80":{    
       "core":"testNodeLost_shard8_replica_n80",           "leader":"true",     
      "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10013_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         "core_node72":{ 
          "core":"testNodeLost_shard8_replica_n72",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10025_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node71":{           
"core":"testNodeLost_shard8_replica_n71",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10088_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node74":{           
"core":"testNodeLost_shard8_replica_n74",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10078_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node73":{           
"core":"testNodeLost_shard8_replica_n73",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10080_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node76":{           
"core":"testNodeLost_shard8_replica_n76",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10059_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node75":{           
"core":"testNodeLost_shard8_replica_n75",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10090_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node78":{           
"core":"testNodeLost_shard8_replica_n78",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10015_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node77":{           
"core":"testNodeLost_shard8_replica_n77",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10006_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node79":{           
"core":"testNodeLost_shard8_replica_n79",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10043_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"d9990000-e665ffff",       
"state":"active"},     "shard9":{       "replicas":{         "core_node90":{    
       "core":"testNodeLost_shard9_replica_n90",           "leader":"true",     
      "SEARCHER.searcher.maxDoc":0,           
"SEARCHER.searcher.deletedDocs":0,           "INDEX.sizeInBytes":1,           
"node_name":"127.0.0.1:10000_solr",           "state":"active",           
"type":"NRT",           "SEARCHER.searcher.numDocs":0},         
"core_node228":{           "node_name":"127.0.0.1:10050_solr",           
"core":"testNodeLost_shard9_replica_n227",           "state":"active",          
 "INDEX.sizeInBytes":1,           "type":"NRT"},         "core_node81":{        
   "core":"testNodeLost_shard9_replica_n81",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10030_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node83":{           
"core":"testNodeLost_shard9_replica_n83",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10081_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node256":{           
"node_name":"127.0.0.1:10000_solr",           
"core":"testNodeLost_shard9_replica_n255",           "state":"active",          
 "INDEX.sizeInBytes":1,           "type":"NRT"},         "core_node84":{        
   "core":"testNodeLost_shard9_replica_n84",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10095_solr",       
    "state":"down",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node87":{           
"core":"testNodeLost_shard9_replica_n87",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10054_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node232":{           
"node_name":"127.0.0.1:10008_solr",           
"core":"testNodeLost_shard9_replica_n231",           "state":"active",          
 "INDEX.sizeInBytes":1,           "type":"NRT"},         "core_node89":{        
   "core":"testNodeLost_shard9_replica_n89",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10042_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0},         "core_node88":{           
"core":"testNodeLost_shard9_replica_n88",           
"SEARCHER.searcher.maxDoc":0,           "SEARCHER.searcher.deletedDocs":0,      
     "INDEX.sizeInBytes":1,           "node_name":"127.0.0.1:10003_solr",       
    "state":"active",           "type":"NRT",           
"SEARCHER.searcher.numDocs":0}},       "range":"e6660000-f332ffff",       
"state":"active"}}}

Stack Trace:
java.util.concurrent.TimeoutException: last state: 
DocCollection(testNodeLost//clusterstate.json/93)={
  "replicationFactor":"10",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"5",
  "autoAddReplicas":"false",
  "nrtReplicas":"10",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "shards":{
    "shard20":{
      "replicas":{
        "core_node191":{
          "core":"testNodeLost_shard20_replica_n191",
          "leader":"true",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10011_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node258":{
          "node_name":"127.0.0.1:10009_solr",
          "core":"testNodeLost_shard20_replica_n257",
          "state":"active",
          "INDEX.sizeInBytes":1,
          "type":"NRT"},
        "core_node200":{
          "core":"testNodeLost_shard20_replica_n200",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10028_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node266":{
          "node_name":"127.0.0.1:10028_solr",
          "core":"testNodeLost_shard20_replica_n265",
          "state":"active",
          "INDEX.sizeInBytes":1,
          "type":"NRT"},
        "core_node199":{
          "core":"testNodeLost_shard20_replica_n199",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10074_solr",
          "state":"active",
          "type":"NRT",
          "SEARCHER.searcher.numDocs":0},
        "core_node196":{
          "core":"testNodeLost_shard20_replica_n196",
          "SEARCHER.searcher.maxDoc":0,
          "SEARCHER.searcher.deletedDocs":0,
          "INDEX.sizeInBytes":1,
          "node_name":"127.0.0.1:10071_solr",
          "state":"active",
          "type":"NRT",
         

[...truncated too long message...]

d-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/null1287052585
     [copy] Copying 39 files to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/null1287052585
   [delete] Deleting directory 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/null1287052585

resolve-example:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

resolve-server:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/null1656161918
     [copy] Copying 246 files to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/null1656161918
   [delete] Deleting directory 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-7.x/solr/null1656161918

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: 
org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath]         confs: [default]
[ivy:cachepath]         found 
org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath]         found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath]         found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath]         found org.apache.httpcomponents#httpclient;4.3.6 in 
public
[ivy:cachepath]         found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath]         found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath]         found commons-codec#commons-codec;1.6 in public
[ivy:cachepath]         found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 64ms :: artifacts dl 9ms
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for 
further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 275 minutes 48 seconds
Archiving artifacts
WARN: No artifacts found that match the file pattern 
"**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
WARN: java.lang.InterruptedException: no matches found within 10000
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Reply via email to